Oct 31 02:54:40.171344 ip-10-0-139-52 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:54:40.564165 ip-10-0-139-52 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:40.564165 ip-10-0-139-52 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:54:40.564165 ip-10-0-139-52 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:40.564165 ip-10-0-139-52 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:54:40.568703 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568622 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.568703 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568702 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.568703 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568706 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568710 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568713 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568715 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568718 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568721 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568724 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568728 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568736 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568739 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568742 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568745 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568747 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568750 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568753 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568755 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568758 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568761 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568763 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568766 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.568804 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568769 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568771 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568774 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568778 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568782 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568785 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568788 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568791 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568795 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568799 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568802 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568805 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568808 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568811 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568814 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568817 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568820 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568822 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568826 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.569465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568828 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.570083 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568831 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.570083 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.568878 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.571210 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571196 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.571210 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571209 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571213 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571217 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571222 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571226 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571229 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571232 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571235 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571238 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571242 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571244 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571272 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571275 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571278 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.571278 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571282 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571295 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571325 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571476 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571482 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571487 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571490 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571494 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571496 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571499 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571502 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571504 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571507 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571510 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571512 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571515 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571519 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571521 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571525 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.571631 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571528 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571531 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571541 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571546 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571549 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571552 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571555 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571557 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571560 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571563 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571568 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571571 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571573 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571576 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571579 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571581 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571585 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571588 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571591 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.572110 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571595 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571597 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571600 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571603 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571607 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571610 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571616 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571620 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.571623 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572481 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572488 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572492 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572495 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572499 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572502 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572505 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572507 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572510 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572513 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572515 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.572561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572518 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572521 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572524 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572526 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572529 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572532 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572535 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572537 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572540 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572543 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572545 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572548 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572550 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572553 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572557 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572560 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572563 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572566 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572568 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572572 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.573085 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572575 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572578 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572581 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572584 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572586 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572589 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572592 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572595 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572599 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572604 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572607 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572610 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572613 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572616 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572619 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572621 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572624 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572626 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572629 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572631 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.573565 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572634 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572636 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572640 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572643 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572645 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572648 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572652 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572655 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572657 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572660 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572663 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572665 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572668 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572671 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572673 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572676 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572678 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572681 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572683 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.574915 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572686 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572688 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572691 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572694 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572696 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572699 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572701 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572703 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572706 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572708 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572711 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572713 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572716 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572718 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572721 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572724 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572727 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572730 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572732 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572735 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.575387 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572738 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572741 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572743 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572746 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572748 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572751 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572754 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572756 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572759 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572761 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572764 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572769 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572772 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572775 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572778 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.572780 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573291 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573296 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.575952 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573300 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573303 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573305 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573308 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573312 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573314 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573317 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573320 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573322 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573325 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573328 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573331 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573333 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573337 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573339 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573342 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573345 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573348 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573350 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573353 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.576393 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573356 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573358 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573361 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573363 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573366 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573369 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573371 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573374 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573377 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573379 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573382 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573385 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573387 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573390 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573392 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573395 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573398 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573401 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573404 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.576863 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573407 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573411 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573415 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573418 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573422 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573424 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573427 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573429 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573432 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573434 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573437 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573439 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573443 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573445 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573448 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573451 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573453 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573456 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573458 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573461 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.577325 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573464 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573466 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573469 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573472 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573474 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573477 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573479 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573482 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573484 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573488 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573490 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573492 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573495 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573498 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573500 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573503 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573507 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573511 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573514 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573517 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.577803 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573520 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573522 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573525 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573527 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573530 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573532 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573535 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573537 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573540 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573542 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573544 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573547 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573549 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573552 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573555 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573558 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573561 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573563 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573566 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.578291 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573568 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573571 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573574 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573577 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573579 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.573582 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575503 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575520 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575527 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575532 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575549 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575554 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575559 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575563 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575567 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575570 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575574 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575577 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575580 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575583 2583 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575586 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575590 2583 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575593 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:54:40.578769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575596 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575602 2583 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575605 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575608 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575612 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575615 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575620 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575623 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575627 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575630 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575633 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575637 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575640 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575643 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575646 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575650 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575653 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575656 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575659 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575663 2583 flags.go:64] FLAG: --enable-server="true" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575666 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575671 2583 flags.go:64] FLAG: --event-burst="100" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575674 2583 flags.go:64] FLAG: --event-qps="50" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575678 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:54:40.579324 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575681 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575684 2583 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575688 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575691 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575694 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575698 2583 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575701 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575703 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575707 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575710 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575712 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575715 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575718 2583 flags.go:64] FLAG: --feature-gates="" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575723 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575726 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575729 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575732 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575735 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575738 2583 flags.go:64] FLAG: --help="false" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575741 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575745 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575748 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575751 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:54:40.579938 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575754 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575758 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575761 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575764 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575769 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575773 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575776 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575779 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575783 2583 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575786 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575788 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575792 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575794 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575798 2583 flags.go:64] FLAG: --lock-file="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575801 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575804 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575807 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575813 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575816 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575819 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575822 2583 flags.go:64] FLAG: --logging-format="text" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575825 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575828 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575831 2583 flags.go:64] FLAG: --manifest-url="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575834 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:54:40.580499 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575839 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575842 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575846 2583 flags.go:64] FLAG: --max-pods="110" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575851 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575855 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575858 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575861 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575864 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575867 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575870 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575877 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575881 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575885 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575888 2583 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575891 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575895 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575898 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575901 2583 flags.go:64] FLAG: --port="10250" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575904 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575907 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-075d65ea86d0e6a73" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575910 2583 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575913 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575917 2583 flags.go:64] FLAG: --register-node="true" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575920 2583 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575924 2583 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575927 2583 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:54:40.581106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575929 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575932 2583 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575936 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575939 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575942 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575945 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575948 2583 flags.go:64] FLAG: --runonce="false" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575951 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575954 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575959 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575962 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575966 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575969 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575972 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575975 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575978 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575981 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575985 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575988 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575992 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575995 2583 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.575998 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576004 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576007 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576010 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:54:40.581730 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576014 2583 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576017 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576020 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576023 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576026 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576029 2583 flags.go:64] FLAG: --v="2" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576044 2583 flags.go:64] FLAG: --version="false" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576049 2583 flags.go:64] FLAG: --vmodule="" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576053 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576056 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576196 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576200 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576203 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576206 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576208 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576211 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576215 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576218 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576221 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576224 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576226 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.582410 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576229 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576232 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576234 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576239 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576241 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576244 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576247 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576250 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576252 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576256 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576258 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576261 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576264 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576267 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576269 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576272 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576275 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576278 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576280 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.582910 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576283 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576285 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576288 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576291 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576293 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576296 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576299 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576301 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576305 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576308 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576311 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576313 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576316 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576318 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576321 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576324 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576327 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576330 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576333 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576336 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.583375 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576338 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576341 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576343 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576347 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576350 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576353 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576356 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576358 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576361 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576363 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576367 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576369 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576372 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576376 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576379 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576382 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576384 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576387 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576390 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576392 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.583850 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576396 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576399 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576402 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576405 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576408 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576411 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576414 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576416 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576420 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576423 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576426 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576428 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576431 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576434 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576436 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576439 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576442 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576445 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576448 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576450 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.584338 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576453 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576455 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576459 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576464 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576467 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576470 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576473 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576475 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576478 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576481 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576483 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576486 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576490 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576492 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576495 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.584816 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576498 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.576555 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576599 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576605 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576608 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576611 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576615 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576618 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576621 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576623 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576626 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576629 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576631 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576634 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576637 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576640 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.585183 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576643 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576645 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576648 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576651 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576655 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576659 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576662 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576665 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576668 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576670 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576673 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576675 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576678 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576683 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576685 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576688 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576691 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576693 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576696 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.585561 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576699 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576701 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576704 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576708 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576710 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576713 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576716 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576718 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576721 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576724 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576727 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576729 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576732 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576735 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576738 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576740 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576743 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576745 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576748 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.586021 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576750 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576753 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576755 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576758 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576761 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576763 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576766 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576770 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576772 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576775 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576778 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576780 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576783 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576786 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576788 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576791 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576795 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576797 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576800 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576803 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.586493 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576805 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576808 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576811 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576814 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576816 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576818 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576822 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576825 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576827 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576830 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576832 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576835 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576837 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576840 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576843 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576846 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576849 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576851 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576854 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576857 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.586978 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576860 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576863 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576865 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576869 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576873 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576877 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576880 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576882 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576886 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576888 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576891 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576893 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576896 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.576899 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.583862 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.583876 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.583892 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.583897 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583956 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583961 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.587463 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583964 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583967 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583970 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583973 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583976 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583979 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583982 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583984 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583987 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583990 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583992 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583996 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.583998 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584001 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584004 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584006 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584009 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584012 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584015 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584017 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.588090 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584020 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584022 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584025 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584028 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584042 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584046 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584049 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584053 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584057 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584060 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584065 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584069 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584073 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584076 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584082 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584085 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584088 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584090 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.588580 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584093 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584096 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584098 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584101 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584104 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584107 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584109 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584112 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584115 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584118 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584121 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584124 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584126 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584129 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584132 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584134 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584137 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584139 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584142 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584145 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.589170 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584147 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584151 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584154 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584156 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584160 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584163 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584165 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584168 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584170 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584173 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584176 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584179 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584181 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584184 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584186 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584189 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584192 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584195 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584198 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.589666 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584200 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584203 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584206 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584208 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584211 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584213 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584216 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584219 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584221 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584224 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584226 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584229 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584231 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584234 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584236 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584239 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584242 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584244 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584247 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584250 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.590142 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584253 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584255 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584258 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584261 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584264 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584266 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.584293 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584462 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584468 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584471 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584475 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584477 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584481 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584484 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584487 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584490 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.590687 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584493 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584496 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584498 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584501 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584504 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584507 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584509 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584512 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584515 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584517 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584520 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584522 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584525 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584528 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584531 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584533 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584536 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584539 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584541 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.591088 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584544 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584547 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584549 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584552 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584555 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584557 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584560 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584562 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584565 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584568 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584570 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584573 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584575 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584578 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584580 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584583 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584586 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584588 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584591 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584593 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.591699 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584596 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584598 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584601 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584603 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584606 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584608 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584611 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584614 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584617 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584622 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584625 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584628 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584631 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584633 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584636 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584638 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584641 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584643 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584645 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584648 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.592506 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584651 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584653 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584656 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584658 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584661 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584663 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584666 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584668 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584671 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584673 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584676 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584679 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584681 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584684 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584686 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584689 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584691 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584694 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584696 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584700 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.592998 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584702 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584707 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584711 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584715 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584720 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584723 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584726 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584729 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584731 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584734 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584736 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584739 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584741 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584744 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584746 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584749 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584751 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.593560 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584754 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.584778 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584810 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584814 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584817 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584820 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584823 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584826 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584828 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584831 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584833 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584836 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584839 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584841 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584844 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584847 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:54:40.593975 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584851 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584854 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584856 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584859 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584861 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584864 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584866 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584869 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584871 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584874 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584876 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584879 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584881 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584884 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584886 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584889 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584892 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584894 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584897 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584901 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:54:40.594435 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584903 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584906 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584908 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584911 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584914 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584916 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584919 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584921 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584924 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584927 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584931 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584935 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584941 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584944 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584947 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584950 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584953 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584956 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:54:40.594954 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584958 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584961 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584964 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584966 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584969 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584971 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584974 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584976 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584979 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584982 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584984 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584987 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584989 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584993 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584996 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.584999 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585002 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585005 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585008 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:54:40.595465 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585010 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585013 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585015 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585018 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585021 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585023 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585026 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585030 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585043 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585046 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585049 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585052 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585054 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585057 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585060 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585062 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585065 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585068 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585070 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:54:40.596259 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585074 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585076 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585082 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585084 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585087 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585089 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585095 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585097 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585100 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585102 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585105 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585108 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585111 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585114 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:40.585116 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.585170 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.587457 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.588519 2583 server.go:1014] "Starting client certificate rotation" Oct 31 02:54:40.596851 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.588629 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:40.597503 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.588687 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:54:40.597503 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.590686 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:40.597503 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.593645 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:54:40.604668 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.604646 2583 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:54:40.608597 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.608580 2583 log.go:25] "Validated CRI v1 image API" Oct 31 02:54:40.610512 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.610485 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:54:40.613149 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.613130 2583 fs.go:135] Filesystem UUIDs: map[550a4c21-a443-40a6-9537-b32bc2259b88:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fcff9b2c-71a1-4958-bedb-3fce9b2743f2:/dev/nvme0n1p4] Oct 31 02:54:40.613198 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.613149 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 02:54:40.620067 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.619912 2583 manager.go:217] Machine: {Timestamp:2026-10-31 02:54:40.618481646 +0000 UTC m=+0.350763358 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132808 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e3f7be11e4f6a83ac63ea20a5981 SystemUUID:ec22e3f7-be11-e4f6-a83a-c63ea20a5981 BootID:6ba96047-ef76-4e7f-9494-236e190692f7 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:7a:5f:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:7a:5f:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:a1:65:7a:b6:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:54:40.620067 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.620065 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:54:40.620181 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.620153 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:54:40.621396 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621368 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:54:40.621553 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621399 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:54:40.621601 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621555 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:54:40.621601 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621564 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:54:40.621601 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621599 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:40.621678 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.621617 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:54:40.622923 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.622913 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:54:40.623199 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.623181 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:40.627603 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627591 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:54:40.627732 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627723 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:54:40.627767 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627757 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:54:40.627794 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627770 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:54:40.627794 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627783 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:54:40.627794 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627792 2583 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:54:40.627897 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.627810 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:54:40.628166 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.628155 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:40.632895 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.632876 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:54:40.639978 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.639965 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:54:40.640868 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.640856 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:54:40.640904 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.640888 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:54:40.642192 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642181 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:54:40.642234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642198 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:54:40.642234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642204 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:54:40.642234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642210 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:54:40.642234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642216 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:54:40.642234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.642222 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:54:40.643208 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643199 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:54:40.643244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643210 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:54:40.643244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643217 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:54:40.643244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643223 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:54:40.643244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643232 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:54:40.643244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.643241 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:54:40.644986 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.644970 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:54:40.645063 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.644994 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:54:40.648807 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.648777 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dql45" Oct 31 02:54:40.650652 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.650633 2583 server.go:1263] "Started kubelet" Oct 31 02:54:40.650771 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.650716 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:54:40.650880 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.650823 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:54:40.650949 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.650906 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:54:40.651466 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.651441 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 02:54:40.651573 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.651479 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 02:54:40.651608 ip-10-0-139-52 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:54:40.652029 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.651678 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 02:54:40.652365 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.652347 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:54:40.654707 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.654330 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:54:40.656387 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.656361 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 02:54:40.656971 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.656957 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dql45" Oct 31 02:54:40.661468 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.660369 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-52.us-east-2.compute.internal.18e37db221efaba3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 02:54:40.650595235 +0000 UTC m=+0.382876945,LastTimestamp:2026-10-31 02:54:40.650595235 +0000 UTC m=+0.382876945,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-52.us-east-2.compute.internal,}" Oct 31 02:54:40.661758 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.661741 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:54:40.661848 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.661754 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:40.661848 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.661826 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:40.662488 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662473 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:54:40.662488 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662489 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:54:40.662625 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662475 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:54:40.662625 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662591 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:54:40.662625 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662617 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:54:40.662818 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662798 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:54:40.662818 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662818 2583 factory.go:55] Registering systemd factory Oct 31 02:54:40.662955 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662827 2583 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:54:40.663015 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.662997 2583 factory.go:153] Registering CRI-O factory Oct 31 02:54:40.663015 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.663017 2583 factory.go:223] Registration of the crio container factory successfully Oct 31 02:54:40.663156 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.663057 2583 factory.go:103] Registering Raw factory Oct 31 02:54:40.663156 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.663074 2583 manager.go:1199] Started watching for new ooms in manager Oct 31 02:54:40.664077 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.664058 2583 manager.go:319] Starting recovery of all containers Oct 31 02:54:40.664446 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.664421 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 02:54:40.669885 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.669865 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:40.677809 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.677761 2583 manager.go:324] Recovery completed Oct 31 02:54:40.682224 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.682211 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.684171 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684153 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.684252 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684182 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.684252 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684195 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.684769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684756 2583 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:54:40.684769 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684767 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:54:40.684849 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684783 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:54:40.684849 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.684821 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.686508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686495 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.686583 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686519 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.686583 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686531 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.686795 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686784 2583 policy_none.go:50] "Start" Oct 31 02:54:40.686837 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686801 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:54:40.686837 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.686811 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:54:40.688794 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.688780 2583 policy_none.go:44] "Start" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.728720 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.731569 2583 manager.go:341] "Starting Device Plugin manager" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.731593 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.731602 2583 server.go:87] "Starting device plugin registration server" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.731985 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.731997 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.732220 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.732297 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.732305 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.732793 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.732839 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.737796 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.738926 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.738949 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.738967 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.738997 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 02:54:40.743963 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.743458 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:40.832685 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.832599 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.834285 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.834269 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.834389 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.834300 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.834389 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.834322 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.834389 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.834354 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.839608 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.839585 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal"] Oct 31 02:54:40.839670 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.839648 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.840590 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.840575 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.840657 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.840604 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.840657 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.840617 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.842007 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.841991 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.842067 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.842014 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-52.us-east-2.compute.internal\": node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:40.842785 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.842772 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.842932 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.842918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.842968 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.842948 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.843608 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843581 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.843713 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843620 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.843713 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843629 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.843713 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843581 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.843713 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843699 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.843713 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.843710 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.845663 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.845647 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.845732 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.845676 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:54:40.846471 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.846454 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:54:40.846566 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.846480 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:54:40.846566 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.846491 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:54:40.858984 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.858836 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:40.867569 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.867547 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.869633 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.869615 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.959475 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:40.959444 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:40.963761 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.963742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.963830 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.963772 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:40.963830 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:40.963813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.059610 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:41.059578 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:41.064932 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.064906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.064992 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.064939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.064992 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.064956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.064992 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.064982 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.065106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.065011 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.065106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.065014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.160391 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:41.160358 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:41.173882 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.173858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.173933 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:41.173881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:41.260779 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:41.260747 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.421658 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.421624 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.522175 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.522100 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.622641 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.622607 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.648974 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.648941 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 02:54:42.649476 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.649080 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 02:54:42.649476 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.649130 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 02:54:42.724241 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.724191 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.724378 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.724222 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:41 +0000 UTC" deadline="2028-06-06 15:45:39.755416237 +0000 UTC" Oct 31 02:54:42.724378 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.724261 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14028h50m57.031157279s" Oct 31 02:54:42.824484 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.824445 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.921032 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:42.921002 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9366a4079c21e3a2e7683bc4a50b18.slice/crio-2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18 WatchSource:0}: Error finding container 2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18: Status 404 returned error can't find the container with id 2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18 Oct 31 02:54:42.921662 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:42.921638 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b1495007bcae44e02a368d0e145561.slice/crio-1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188 WatchSource:0}: Error finding container 1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188: Status 404 returned error can't find the container with id 1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188 Oct 31 02:54:42.924552 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:42.924530 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:42.927147 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:42.927128 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:54:43.025013 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.024968 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:43.042652 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.042581 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:43.125756 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.125721 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:43.226554 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.226520 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:54:43.233851 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.233831 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:43.322971 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.322885 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:43.335880 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.335855 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:43.337064 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.337045 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:54:43.346213 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.346192 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:54:43.690538 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.690283 2583 apiserver.go:52] "Watching apiserver" Oct 31 02:54:43.701670 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.701641 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:54:43.701990 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.701970 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k45bw","openshift-multus/multus-6t7rb","kube-system/global-pull-secret-syncer-j8dxn","openshift-network-diagnostics/network-check-target-5q2kq","openshift-ovn-kubernetes/ovnkube-node-7ngch","kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-mhr97","openshift-network-operator/iptables-alerter-2zf5j","kube-system/konnectivity-agent-chd4v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw","openshift-cluster-node-tuning-operator/tuned-9pj75","openshift-image-registry/node-ca-wjn6d","openshift-multus/network-metrics-daemon-29cdk"] Oct 31 02:54:43.704688 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.704437 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.707823 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.707798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.708897 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.708869 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:54:43.708897 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.708889 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:54:43.709056 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.709005 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:54:43.711233 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.711215 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.711317 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.711275 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.714056 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.714135 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.714374 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.714383 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.714375 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:54:43.714659 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.714487 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.716340 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.716319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.721316 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.721298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.723743 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.723724 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 02:54:43.724365 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.724345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.726305 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.726283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.734803 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.734783 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.734982 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.734962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735056 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.734997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm24j\" (UniqueName: \"kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.735056 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.735140 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.735197 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.735254 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735239 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735305 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:43.735305 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735427 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735427 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735427 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.735575 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.735575 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.735575 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.735683 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735683 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735683 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735655 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735797 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735797 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.735797 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbv4\" (UniqueName: \"kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.735797 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735967 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735802 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.735967 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.735934 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736079 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljwr\" (UniqueName: \"kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736135 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.736135 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb8c\" (UniqueName: \"kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.736236 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736236 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736210 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736331 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736331 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736331 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.736493 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736348 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.736493 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736372 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.736493 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736493 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csml\" (UniqueName: \"kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.736662 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.736662 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.736662 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.736571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.738779 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.737133 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.739600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.739578 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.740340 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.740121 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 02:54:43.741670 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.741648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.741752 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.741719 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:43.764015 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.763989 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgrnk" Oct 31 02:54:43.770703 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.770681 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgrnk" Oct 31 02:54:43.804907 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.804849 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"01b1495007bcae44e02a368d0e145561","Type":"ContainerStarted","Data":"1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188"} Oct 31 02:54:43.806205 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.806177 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerStarted","Data":"2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18"} Oct 31 02:54:43.809409 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809326 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.809409 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809326 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:54:43.809409 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809355 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.809409 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809377 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:54:43.809409 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809384 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:54:43.809750 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.809717 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:54:43.810175 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.810160 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.810261 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.810164 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:54:43.811056 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811038 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.811056 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811038 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.811207 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811045 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:54:43.811207 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811072 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.811602 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811563 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:54:43.811932 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811908 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:54:43.811932 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811932 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.812062 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811978 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:54:43.812062 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.811982 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.812062 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.812009 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:54:43.812062 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.812026 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:54:43.812062 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.812047 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:54:43.812326 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.812064 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.812326 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.812097 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:54:43.813435 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813402 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.813521 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813464 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:54:43.813621 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813602 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:54:43.813737 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813648 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:54:43.813737 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813651 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:54:43.813737 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813698 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:54:43.813913 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.813769 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:54:43.824233 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.824210 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:54:43.837084 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837051 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb8c\" (UniqueName: \"kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837095 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6ln\" (UniqueName: \"kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837222 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837241 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm24j\" (UniqueName: \"kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837508 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837570 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837590 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837612 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837745 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.837867 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837844 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.837996 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838058 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838094 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838122 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838229 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.838530 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838255 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838296 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lzn\" (UniqueName: \"kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838375 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8xg\" (UniqueName: \"kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838721 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839189 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838865 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838920 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838954 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.838955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839005 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csml\" (UniqueName: \"kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839196 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.839753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839397 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839571 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839606 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839666 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.839980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840062 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.840408 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvkt\" (UniqueName: \"kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840156 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljwr\" (UniqueName: \"kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840341 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840368 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbv4\" (UniqueName: \"kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.840504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.840559 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:43.841187 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.840733 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.340712634 +0000 UTC m=+3.012614969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:43.841852 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.841735 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.841852 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.841799 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.843970 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.843947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.844081 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.843986 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:43.849612 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.849330 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:43.849612 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.849357 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:43.849612 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.849372 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:43.849612 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.849445 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.349426411 +0000 UTC m=+3.021328748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:43.850035 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.850009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb8c\" (UniqueName: \"kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:43.850171 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.850154 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm24j\" (UniqueName: \"kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:43.851777 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.851496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csml\" (UniqueName: \"kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:43.854830 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.854810 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbv4\" (UniqueName: \"kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:43.854830 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.854821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljwr\" (UniqueName: \"kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:43.941315 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941315 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941333 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941370 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941514 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6ln\" (UniqueName: \"kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.941536 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:43.941587 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:54:44.441568077 +0000 UTC m=+3.113470402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941585 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941605 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.941835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941904 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941937 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.941983 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942052 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942078 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lzn\" (UniqueName: \"kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.942217 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8xg\" (UniqueName: \"kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942371 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942025 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942484 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942493 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942534 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942547 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943021 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942636 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942707 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942732 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942732 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942763 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942814 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942841 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:43.943600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942891 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942900 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942906 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942936 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.942977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.943017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.943075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvkt\" (UniqueName: \"kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.943073 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.943208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.944170 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.943260 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.944502 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.944336 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.944502 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.944458 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.954989 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.954960 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6ln\" (UniqueName: \"kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:43.955468 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.955444 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8xg\" (UniqueName: \"kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:43.955571 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.955548 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvkt\" (UniqueName: \"kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:54:43.955611 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:43.955570 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lzn\" (UniqueName: \"kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:44.020855 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.020813 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:44.031488 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.031461 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:54:44.031797 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.031770 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8246785f_965d_4107_9e6e_1bed982b3d0e.slice/crio-1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5 WatchSource:0}: Error finding container 1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5: Status 404 returned error can't find the container with id 1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5 Oct 31 02:54:44.040535 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.040503 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd57deac_c763_4eb7_9010_8d38ea97766c.slice/crio-026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66 WatchSource:0}: Error finding container 026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66: Status 404 returned error can't find the container with id 026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66 Oct 31 02:54:44.060352 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.060319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:54:44.064066 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.064042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:54:44.070163 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.070138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:54:44.072407 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.072378 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc5c3ae_2635_4f11_a2fa_cc14fbb02db7.slice/crio-a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e WatchSource:0}: Error finding container a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e: Status 404 returned error can't find the container with id a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e Oct 31 02:54:44.072841 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.072822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:54:44.074611 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.074586 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c89547_d1bb_4dfa_94e5_cd303ed05f2a.slice/crio-5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59 WatchSource:0}: Error finding container 5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59: Status 404 returned error can't find the container with id 5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59 Oct 31 02:54:44.075399 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.075385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6t7rb" Oct 31 02:54:44.081043 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.081017 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3abcfe_0522_4dfb_abda_7f0707b65a65.slice/crio-f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4 WatchSource:0}: Error finding container f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4: Status 404 returned error can't find the container with id f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4 Oct 31 02:54:44.083846 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.083816 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe728edf_d50f_4c7b_b98c_a12c19c39fa6.slice/crio-4f3700a9e9a7b498bcba07f74a7e8e81a5313f046ac3ec1adfd6f252fd68d157 WatchSource:0}: Error finding container 4f3700a9e9a7b498bcba07f74a7e8e81a5313f046ac3ec1adfd6f252fd68d157: Status 404 returned error can't find the container with id 4f3700a9e9a7b498bcba07f74a7e8e81a5313f046ac3ec1adfd6f252fd68d157 Oct 31 02:54:44.087176 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.087155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:54:44.087403 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.087369 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833330df_2c89_48b4_9c1d_c049b3378e26.slice/crio-b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9 WatchSource:0}: Error finding container b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9: Status 404 returned error can't find the container with id b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9 Oct 31 02:54:44.089473 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.089456 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k45bw" Oct 31 02:54:44.097641 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.097616 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d8733a_ce73_455e_aeb6_bba45d34b97f.slice/crio-ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582 WatchSource:0}: Error finding container ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582: Status 404 returned error can't find the container with id ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582 Oct 31 02:54:44.099993 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:54:44.099969 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371390b5_879d_403c_b7e5_58bbceb94ba0.slice/crio-b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e WatchSource:0}: Error finding container b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e: Status 404 returned error can't find the container with id b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e Oct 31 02:54:44.217988 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.217914 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:54:44.345714 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.345631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:44.345892 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.345745 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:44.345892 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.345810 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:54:45.345793597 +0000 UTC m=+4.017695935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.445978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.446056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446130 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446153 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446165 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446171 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446223 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:54:45.44620623 +0000 UTC m=+4.118108551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:44.446227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:44.446241 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:54:45.446233666 +0000 UTC m=+4.118135985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:44.773855 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.773664 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:43 +0000 UTC" deadline="2028-05-25 13:14:00.853065553 +0000 UTC" Oct 31 02:54:44.773855 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.773727 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13738h19m16.079342307s" Oct 31 02:54:44.810279 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.810163 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t7rb" event={"ID":"833330df-2c89-48b4-9c1d-c049b3378e26","Type":"ContainerStarted","Data":"b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9"} Oct 31 02:54:44.811806 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.811715 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zf5j" event={"ID":"12c89547-d1bb-4dfa-94e5-cd303ed05f2a","Type":"ContainerStarted","Data":"5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59"} Oct 31 02:54:44.814000 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.813964 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e"} Oct 31 02:54:44.817193 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.816486 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"01b1495007bcae44e02a368d0e145561","Type":"ContainerStarted","Data":"eaef77661445882027f2268d6d34a9720dd0db7949282e29287b10dac683db2e"} Oct 31 02:54:44.820593 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.820523 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k45bw" event={"ID":"371390b5-879d-403c-b7e5-58bbceb94ba0","Type":"ContainerStarted","Data":"b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e"} Oct 31 02:54:44.823502 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.823377 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" event={"ID":"c1d8733a-ce73-455e-aeb6-bba45d34b97f","Type":"ContainerStarted","Data":"ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582"} Oct 31 02:54:44.828154 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.828084 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9pj75" event={"ID":"fe728edf-d50f-4c7b-b98c-a12c19c39fa6","Type":"ContainerStarted","Data":"4f3700a9e9a7b498bcba07f74a7e8e81a5313f046ac3ec1adfd6f252fd68d157"} Oct 31 02:54:44.835813 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.835273 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" podStartSLOduration=1.835257963 podStartE2EDuration="1.835257963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:44.834358371 +0000 UTC m=+3.506260728" watchObservedRunningTime="2026-10-31 02:54:44.835257963 +0000 UTC m=+3.507160309" Oct 31 02:54:44.838476 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.838226 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerStarted","Data":"f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4"} Oct 31 02:54:44.841756 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.841361 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjn6d" event={"ID":"dd57deac-c763-4eb7-9010-8d38ea97766c","Type":"ContainerStarted","Data":"026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66"} Oct 31 02:54:44.844836 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:44.844772 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chd4v" event={"ID":"8246785f-965d-4107-9e6e-1bed982b3d0e","Type":"ContainerStarted","Data":"1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5"} Oct 31 02:54:45.360098 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.359869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:45.360286 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.360225 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:45.360345 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.360294 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.360276164 +0000 UTC m=+6.032178486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.460987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.461066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461197 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461255 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.461240118 +0000 UTC m=+6.133142461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461699 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461742 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461754 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:45.461835 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.461797 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:54:47.461783701 +0000 UTC m=+6.133686041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:45.774645 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.774599 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:43 +0000 UTC" deadline="2028-06-17 20:35:37.702021848 +0000 UTC" Oct 31 02:54:45.774645 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.774644 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14297h40m51.927383367s" Oct 31 02:54:45.802885 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.802841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:45.803035 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.802965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:45.803250 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.803236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:45.803317 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.803305 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:45.803355 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.803345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:45.803401 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:45.803389 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:45.860553 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.860512 2583 generic.go:356] "Generic (PLEG): container finished" podID="ce9366a4079c21e3a2e7683bc4a50b18" containerID="02665be90f8ef455efa93828ffde36ea5c2426afad6ff2350a0bf7eeea3f6242" exitCode=0 Oct 31 02:54:45.861485 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:45.861459 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerDied","Data":"02665be90f8ef455efa93828ffde36ea5c2426afad6ff2350a0bf7eeea3f6242"} Oct 31 02:54:46.873935 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:46.872879 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerStarted","Data":"8d44d426ce79cb76ccf2042113c81a701b73869c12a000efbbc5bd12378f62da"} Oct 31 02:54:46.880920 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:46.880316 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerStarted","Data":"064836b463f3722ca882deb15696f2fe878fb938fb0fff3b2e75d0d4e5b8db9f"} Oct 31 02:54:46.933689 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:46.932838 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" podStartSLOduration=3.932816212 podStartE2EDuration="3.932816212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:46.904673864 +0000 UTC m=+5.576576211" watchObservedRunningTime="2026-10-31 02:54:46.932816212 +0000 UTC m=+5.604718555" Oct 31 02:54:47.382589 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.382088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:47.382589 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.382220 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:47.382589 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.382280 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:54:51.382267111 +0000 UTC m=+10.054169431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.483105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.483178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483333 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483401 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:54:51.483382651 +0000 UTC m=+10.155284995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483519 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483538 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483551 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:47.483928 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.483590 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:54:51.483576712 +0000 UTC m=+10.155479055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.802361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.802510 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.802580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.802646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.802694 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:47.803206 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:47.802762 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:47.886435 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.886305 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="064836b463f3722ca882deb15696f2fe878fb938fb0fff3b2e75d0d4e5b8db9f" exitCode=0 Oct 31 02:54:47.886435 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.886384 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"064836b463f3722ca882deb15696f2fe878fb938fb0fff3b2e75d0d4e5b8db9f"} Oct 31 02:54:47.889789 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:47.889767 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chd4v" event={"ID":"8246785f-965d-4107-9e6e-1bed982b3d0e","Type":"ContainerStarted","Data":"879e5a62bef73e8eb4647f8ab61efae802913e6164a5eb62232cd4a358f6aff9"} Oct 31 02:54:49.802467 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.802086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:49.802467 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:49.802209 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:49.802987 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.802570 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:49.805285 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.802578 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:49.809617 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:49.809158 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:49.809617 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:49.809264 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:49.897027 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.896985 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" event={"ID":"c1d8733a-ce73-455e-aeb6-bba45d34b97f","Type":"ContainerStarted","Data":"6457fe56737c0cb81e6a30802fad55fc31d3e8e24e4bf4cfed27a6544618b6a9"} Oct 31 02:54:49.900080 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.900028 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wjn6d" event={"ID":"dd57deac-c763-4eb7-9010-8d38ea97766c","Type":"ContainerStarted","Data":"fed10da3dba865ad42cfb9a35983749331dfcc951fd92453991799c906a30d21"} Oct 31 02:54:49.918987 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.918927 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-chd4v" podStartSLOduration=5.287274021 podStartE2EDuration="8.918905328s" totalImagesPullingTime="3.631631307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:47.933160079 +0000 UTC m=+6.605062424" watchObservedRunningTime="2026-10-31 02:54:49.918905328 +0000 UTC m=+8.590807666" Oct 31 02:54:49.919343 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:49.919311 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wjn6d" podStartSLOduration=4.17027888 podStartE2EDuration="8.91930026s" totalImagesPullingTime="4.74902138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:49.919060226 +0000 UTC m=+8.590962564" watchObservedRunningTime="2026-10-31 02:54:49.91930026 +0000 UTC m=+8.591202604" Oct 31 02:54:50.953277 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:50.953222 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:54:51.289878 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.289316 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:51.289878 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.289759 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:51.419736 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.418515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:51.419736 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.418686 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:51.419736 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.418747 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:54:59.418730252 +0000 UTC m=+18.090632586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.519565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.519630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.519763 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.519858 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:54:59.519810412 +0000 UTC m=+18.191712732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.520279 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.520297 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.520309 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:51.520390 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.520354 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:54:59.520339779 +0000 UTC m=+18.192242113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:51.802532 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.801652 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:54:50.953240898Z","UUID":"7f2b75bf-4a41-433a-a6eb-ed15428b4b87","Handler":null,"Name":"","Endpoint":""} Oct 31 02:54:51.802532 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.801755 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:51.803227 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.803056 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:51.803227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.803066 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:51.803227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.803135 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:51.803227 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.803151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:51.803227 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:51.803201 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:51.803973 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.803571 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:54:51.803973 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.803601 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:54:51.916607 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.914522 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k45bw" event={"ID":"371390b5-879d-403c-b7e5-58bbceb94ba0","Type":"ContainerStarted","Data":"04e0642f7235bf3395dc70ee92904b6b807ba986c97080f450f7304f2abbe4c5"} Oct 31 02:54:51.916931 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.916904 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" event={"ID":"c1d8733a-ce73-455e-aeb6-bba45d34b97f","Type":"ContainerStarted","Data":"758fa3505107bbf3b5aa1e4285a6ad7cb3cd3bcc4bccf124e03295ac42c0ab23"} Oct 31 02:54:51.921392 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.921322 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6t7rb" event={"ID":"833330df-2c89-48b4-9c1d-c049b3378e26","Type":"ContainerStarted","Data":"34eafd694109fa7246c63da176ab8b82bb3cccb54c1fffdbb33247ef77637765"} Oct 31 02:54:51.954279 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.954224 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k45bw" podStartSLOduration=2.776000269 podStartE2EDuration="9.954205491s" totalImagesPullingTime="7.178205222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:51.933928616 +0000 UTC m=+10.605830958" watchObservedRunningTime="2026-10-31 02:54:51.954205491 +0000 UTC m=+10.626107835" Oct 31 02:54:51.954823 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:51.954785 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6t7rb" podStartSLOduration=3.363510314 podStartE2EDuration="10.954775503s" totalImagesPullingTime="7.591265189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:51.954093134 +0000 UTC m=+10.625995476" watchObservedRunningTime="2026-10-31 02:54:51.954775503 +0000 UTC m=+10.626677846" Oct 31 02:54:52.926531 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.926332 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" event={"ID":"c1d8733a-ce73-455e-aeb6-bba45d34b97f","Type":"ContainerStarted","Data":"af07cdf20bd5aa2e90e71aba488537d56a39fbdf7c19696f22ea63082467ee73"} Oct 31 02:54:52.928448 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.928400 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="ff13fe50e5b6efce8415bb60dc9df65eb156d3ef71bf4fe237f2b018e04eb024" exitCode=0 Oct 31 02:54:52.928589 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.928448 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"ff13fe50e5b6efce8415bb60dc9df65eb156d3ef71bf4fe237f2b018e04eb024"} Oct 31 02:54:52.930106 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.930083 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2zf5j" event={"ID":"12c89547-d1bb-4dfa-94e5-cd303ed05f2a","Type":"ContainerStarted","Data":"f08a69acc6a50dc14de720e71feaab187f07a14271c01491c91dd661640f6828"} Oct 31 02:54:52.948520 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.948443 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" podStartSLOduration=3.865645799 podStartE2EDuration="11.948409712s" totalImagesPullingTime="8.082763913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:52.947742266 +0000 UTC m=+11.619644609" watchObservedRunningTime="2026-10-31 02:54:52.948409712 +0000 UTC m=+11.620312056" Oct 31 02:54:52.996718 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:52.996658 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2zf5j" podStartSLOduration=5.184490429 podStartE2EDuration="11.996640297s" totalImagesPullingTime="6.812149868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:52.996363455 +0000 UTC m=+11.668265797" watchObservedRunningTime="2026-10-31 02:54:52.996640297 +0000 UTC m=+11.668542641" Oct 31 02:54:53.800466 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:53.800432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:53.800634 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:53.800563 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:53.800634 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:53.800432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:53.800760 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:53.800684 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:53.800760 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:53.800737 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:53.800857 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:53.800796 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:53.935304 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:53.935266 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="c21f21bc3e5fddf298f7c6b818693ec66fb079269430046ade6ffd6d0cba417c" exitCode=0 Oct 31 02:54:53.935515 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:53.935354 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"c21f21bc3e5fddf298f7c6b818693ec66fb079269430046ade6ffd6d0cba417c"} Oct 31 02:54:54.940671 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:54.940636 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9pj75" event={"ID":"fe728edf-d50f-4c7b-b98c-a12c19c39fa6","Type":"ContainerStarted","Data":"993813ee2b304fafd7677fb62f87383d963f3fd02af1b1b7039c4d03523fe615"} Oct 31 02:54:54.960980 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:54.960932 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9pj75" podStartSLOduration=3.295920903 podStartE2EDuration="13.960917077s" totalImagesPullingTime="10.664996174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:54:54.960529278 +0000 UTC m=+13.632431637" watchObservedRunningTime="2026-10-31 02:54:54.960917077 +0000 UTC m=+13.632819419" Oct 31 02:54:55.800089 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:55.800058 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:55.800268 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:55.800058 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:55.800268 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:55.800172 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:55.800268 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:55.800194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:55.800450 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:55.800274 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:55.800450 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:55.800345 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:55.945248 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:55.945218 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="2e600a0cba4b859501a0b003f08425962f32ce4825e6807e7e06f6eaf39c9e7f" exitCode=0 Oct 31 02:54:55.945859 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:55.945321 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"2e600a0cba4b859501a0b003f08425962f32ce4825e6807e7e06f6eaf39c9e7f"} Oct 31 02:54:57.800346 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:57.800306 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:57.800832 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:57.800372 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:57.800832 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:57.800388 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:57.800832 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:57.800485 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:57.800993 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:57.800868 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:57.800993 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:57.800973 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:58.954809 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.954783 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="26c6b8aaf0fe41d243c3471e214fc209e89baf60da13a40ae09634c05f087c24" exitCode=0 Oct 31 02:54:58.955139 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.954880 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"26c6b8aaf0fe41d243c3471e214fc209e89baf60da13a40ae09634c05f087c24"} Oct 31 02:54:58.961532 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.961502 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"f9ad66931cf50c5cd1cba9640ad412ca38cf00f92d559bee2c3247952f4beff1"} Oct 31 02:54:58.961613 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.961591 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"51ce02f4a9529f52a573299f18c75e5f03088534edd1e6166d0450a6134a2fc0"} Oct 31 02:54:58.961650 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.961642 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"f608dc29732cb5f39580160bc31af6aecb7741aa76786ea84a37b29869b51492"} Oct 31 02:54:58.961707 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:58.961690 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"41a038b5284a2c43975852800016f1cbabf93c976a9acc2e3adeaea60c69c094"} Oct 31 02:54:59.478052 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.477884 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:59.478234 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.478165 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:54:59.478602 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.478585 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:54:59.480052 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.480025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:59.480163 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.480153 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:59.480214 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.480204 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:55:15.480191061 +0000 UTC m=+34.152093397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 02:54:59.580924 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.580896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:59.581067 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.581010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:59.581067 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581023 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 02:54:59.581067 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581039 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 02:54:59.581067 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581050 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gq42q for pod openshift-network-diagnostics/network-check-target-5q2kq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:59.581211 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581097 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q podName:636e54a6-ea98-41b1-9f02-d11a89ad344f nodeName:}" failed. No retries permitted until 2026-10-31 02:55:15.581080324 +0000 UTC m=+34.252982658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq42q" (UniqueName: "kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q") pod "network-check-target-5q2kq" (UID: "636e54a6-ea98-41b1-9f02-d11a89ad344f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 02:54:59.581211 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581139 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:59.581211 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.581188 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret podName:b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec nodeName:}" failed. No retries permitted until 2026-10-31 02:55:15.581174525 +0000 UTC m=+34.253076854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret") pod "global-pull-secret-syncer-j8dxn" (UID: "b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec") : object "kube-system"/"original-pull-secret" not registered Oct 31 02:54:59.800311 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.800225 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:54:59.800311 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.800265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:54:59.800557 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.800225 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:54:59.800557 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.800355 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:54:59.800557 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.800460 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:54:59.800695 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:54:59.800560 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:54:59.966428 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.966376 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"9add562578cb09818308b0333d8a23506bf360a70f972ababdfdf539978abd22"} Oct 31 02:54:59.966870 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.966469 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"bf7096931e78475251a8ba2827fd650c7939dde76ec48ae4953f35adafe13654"} Oct 31 02:54:59.969381 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.969351 2583 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="4c93a08b11bc1058dcc80be497bef8ad479d8e9c56ebfafd0340538bd2c5b51e" exitCode=0 Oct 31 02:54:59.969519 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:54:59.969444 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerDied","Data":"4c93a08b11bc1058dcc80be497bef8ad479d8e9c56ebfafd0340538bd2c5b51e"} Oct 31 02:55:00.974308 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:00.974276 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhr97" event={"ID":"cb3abcfe-0522-4dfb-abda-7f0707b65a65","Type":"ContainerStarted","Data":"ce4dc93562154aac5e3ca3190e3cc71435d01e74748d9c94d88162dae24f514c"} Oct 31 02:55:01.003655 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:01.003610 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mhr97" podStartSLOduration=9.868099856 podStartE2EDuration="20.003598974s" totalImagesPullingTime="10.135499118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:01.003134068 +0000 UTC m=+19.675036411" watchObservedRunningTime="2026-10-31 02:55:01.003598974 +0000 UTC m=+19.675501312" Oct 31 02:55:01.800433 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:01.800392 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:01.800600 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:01.800502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:01.800600 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:01.800509 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:01.800685 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:01.800617 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:01.800685 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:01.800664 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:01.800760 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:01.800721 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:01.982005 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:01.981931 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"9886b63eef2dc56c5a9d950aadd5ebf7713e4a86b2a80c42d9dbbe6e5d146b69"} Oct 31 02:55:03.800099 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:03.800074 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:03.800099 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:03.800103 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:03.800801 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:03.800167 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:03.800801 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:03.800248 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:03.800801 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:03.800304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:03.800801 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:03.800381 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:03.989567 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:03.989533 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" event={"ID":"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7","Type":"ContainerStarted","Data":"34ee92c72497a3b5b965fa21e36631da318f74b581af032a1a3bf83d0a021ab7"} Oct 31 02:55:03.989819 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:03.989794 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:04.004960 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.004937 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:04.024940 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.024904 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" podStartSLOduration=8.428056347 podStartE2EDuration="23.024893621s" totalImagesPullingTime="14.596837274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:04.024710232 +0000 UTC m=+22.696612573" watchObservedRunningTime="2026-10-31 02:55:04.024893621 +0000 UTC m=+22.696795964" Oct 31 02:55:04.865777 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.865744 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8dxn"] Oct 31 02:55:04.866145 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.865921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:04.866145 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:04.866021 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:04.869244 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.869212 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29cdk"] Oct 31 02:55:04.869385 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.869274 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5q2kq"] Oct 31 02:55:04.869385 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.869340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:04.869504 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:04.869449 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:04.869538 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.869512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:04.869605 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:04.869587 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:04.991781 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.991755 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:04.992229 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:04.992196 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:05.007799 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:05.007771 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:05.993768 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:05.993741 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:06.800400 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:06.800360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:06.800635 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:06.800360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:06.800635 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:06.800522 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:06.800635 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:06.800359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:06.800635 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:06.800562 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:06.800790 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:06.800645 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:06.995728 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:06.995701 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:08.340492 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.340452 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:08.603664 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.603585 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:08.800166 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.800133 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:08.800166 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.800163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:08.800362 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:08.800235 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:08.800362 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.800265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:08.800362 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:08.800334 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:08.800498 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:08.800372 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:08.999071 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:08.999028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:09.973582 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:09.973545 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:10.104385 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.104354 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:10.413295 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.413248 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:10.547018 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.546940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:10.799623 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.799536 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:10.799623 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.799536 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:10.799623 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:10.799536 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:10.799867 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:10.799651 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8dxn" podUID="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" Oct 31 02:55:10.799867 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:10.799724 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5q2kq" podUID="636e54a6-ea98-41b1-9f02-d11a89ad344f" Oct 31 02:55:10.799867 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:10.799824 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29cdk" podUID="22743d27-b50f-4219-a8b7-198e417780ee" Oct 31 02:55:11.611949 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.611913 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeReady" Oct 31 02:55:11.612319 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.612046 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 02:55:11.675981 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.675951 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7ngz"] Oct 31 02:55:11.726991 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.726962 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6x7z"] Oct 31 02:55:11.727137 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.727057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.732068 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.732036 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:11.732199 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.732095 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:11.732199 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.732166 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:11.732299 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.732285 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:11.743925 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.743904 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7ngz"] Oct 31 02:55:11.744023 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.743938 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6x7z"] Oct 31 02:55:11.744023 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.744010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.756267 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.756247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:11.831729 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.831701 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:11.831729 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.831701 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:11.831901 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.831870 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:11.870515 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.870515 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870474 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.870515 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pswzg\" (UniqueName: \"kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.870690 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.870738 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870705 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqzc\" (UniqueName: \"kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.870775 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.870765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.890838 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.890809 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:11.971402 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971371 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqzc\" (UniqueName: \"kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.971402 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971445 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswzg\" (UniqueName: \"kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:11.971539 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:11.971643 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:11.971614 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:12.471595708 +0000 UTC m=+31.143498028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:11.971931 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.971837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.972101 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.972083 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.975023 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.975002 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:11.984580 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.984552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswzg\" (UniqueName: \"kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:11.984678 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:11.984665 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqzc\" (UniqueName: \"kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:12.050886 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.050852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:12.399289 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.399254 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7ngz"] Oct 31 02:55:12.402458 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:55:12.402425 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191190f6_9b6d_498d_b960_67437ac0b983.slice/crio-42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a WatchSource:0}: Error finding container 42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a: Status 404 returned error can't find the container with id 42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a Oct 31 02:55:12.443688 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.443599 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:12.443857 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.443837 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:12.461154 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.461128 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:12.475951 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.475918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:12.476119 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.476068 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:12.476158 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.476130 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.476114298 +0000 UTC m=+32.148016618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:12.672465 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.672431 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:55:12.700759 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.700683 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkthd"] Oct 31 02:55:12.700910 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.700793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.718320 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.718290 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:55:12.718320 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.718326 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkthd"] Oct 31 02:55:12.718521 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.718368 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.744592 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.744558 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 02:55:12.744753 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.744612 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:12.744902 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.744887 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:12.744961 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.744937 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:12.748697 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.748672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:12.751904 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.751889 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 02:55:12.762933 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.762909 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w5s96\"" Oct 31 02:55:12.763751 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.763740 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:12.763921 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.763907 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:12.800575 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.800546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:12.800720 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.800617 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:12.800720 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.800704 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:12.807525 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.807500 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:12.810313 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.810290 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:12.810454 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.810318 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:12.810454 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.810339 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:12.815348 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.815328 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:12.815443 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.815379 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:12.816452 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.816391 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258"] Oct 31 02:55:12.832114 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.832080 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258"] Oct 31 02:55:12.832263 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.832163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:12.879398 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879366 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.879398 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.879630 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879512 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2tzx\" (UniqueName: \"kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.879630 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.879630 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.879742 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879705 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.879791 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879739 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.879846 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.879878 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.879847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.883186 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.883152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:12.908272 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.908239 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:12.908474 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.908380 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hnld2\"" Oct 31 02:55:12.910397 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.910346 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:12.910525 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.910508 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 02:55:12.981023 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.980945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tzx\" (UniqueName: \"kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981023 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.980995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981207 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981207 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981207 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.981161 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:12.981207 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.981185 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.981377 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.981232 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.48121514 +0000 UTC m=+32.153117460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : configmap "kube-rbac-proxy" not found Oct 31 02:55:12.981377 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:12.981250 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:13.481241024 +0000 UTC m=+32.153143347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:12.981377 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981273 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlb8h\" (UniqueName: \"kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h\") pod \"migrator-66df5d496b-5p258\" (UID: \"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:12.981377 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981559 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981559 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981559 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981559 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981738 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981703 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.981795 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981826 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981779 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.981857 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.981825 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.982338 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.982322 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:12.993880 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.993852 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tzx\" (UniqueName: \"kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:12.993999 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:12.993958 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:13.008605 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.008575 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7ngz" event={"ID":"191190f6-9b6d-498d-b960-67437ac0b983","Type":"ContainerStarted","Data":"42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a"} Oct 31 02:55:13.013869 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.013839 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:13.025160 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.025136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:13.082195 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.082160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlb8h\" (UniqueName: \"kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h\") pod \"migrator-66df5d496b-5p258\" (UID: \"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:13.095380 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.095349 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlb8h\" (UniqueName: \"kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h\") pod \"migrator-66df5d496b-5p258\" (UID: \"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:13.147741 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.147708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:13.333701 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.333530 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:55:13.336351 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:55:13.336319 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503a089d_2343_450d_bba6_486321ca52e4.slice/crio-75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220 WatchSource:0}: Error finding container 75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220: Status 404 returned error can't find the container with id 75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220 Oct 31 02:55:13.441156 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.441124 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258"] Oct 31 02:55:13.445936 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:55:13.445903 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a65a6ce_b6f6_4b43_a4f1_4848a347ce36.slice/crio-c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09 WatchSource:0}: Error finding container c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09: Status 404 returned error can't find the container with id c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09 Oct 31 02:55:13.486172 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.486138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:13.486343 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.486200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:13.486343 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:13.486245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:13.486343 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486322 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:13.486517 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486344 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:13.486517 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486370 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:13.486517 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486395 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:15.486375631 +0000 UTC m=+34.158277967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:13.486517 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486436 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.486405687 +0000 UTC m=+33.158308011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : configmap "kube-rbac-proxy" not found Oct 31 02:55:13.486517 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:13.486453 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:14.486444071 +0000 UTC m=+33.158346394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.013842 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.013734 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" event={"ID":"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36","Type":"ContainerStarted","Data":"c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09"} Oct 31 02:55:14.015200 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.015131 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" event={"ID":"503a089d-2343-450d-bba6-486321ca52e4","Type":"ContainerStarted","Data":"75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220"} Oct 31 02:55:14.501318 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.501280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:14.501318 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.501334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:14.501580 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:14.501439 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 02:55:14.501580 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:14.501470 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.501580 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:14.501499 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.501479966 +0000 UTC m=+35.173382285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : configmap "kube-rbac-proxy" not found Oct 31 02:55:14.501580 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:14.501531 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:16.501513946 +0000 UTC m=+35.173416284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:14.537956 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.537916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:14.644836 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.644792 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jp78g"] Oct 31 02:55:14.678157 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.678122 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jp78g"] Oct 31 02:55:14.678326 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.678267 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.684451 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.684313 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zp7wq\"" Oct 31 02:55:14.685884 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.685859 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 02:55:14.686006 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.685974 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 02:55:14.686109 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.686080 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 02:55:14.686187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.686107 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 02:55:14.695250 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.695225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:14.803187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.803103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.803187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.803143 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.803187 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.803162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkd9q\" (UniqueName: \"kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.904706 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.904671 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.904706 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.904712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.904935 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.904729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkd9q\" (UniqueName: \"kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.905885 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.905863 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.907529 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.907505 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.917990 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.917966 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkd9q\" (UniqueName: \"kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:14.999802 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:14.999769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:15.142589 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.142547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:15.281862 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.281775 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:15.289997 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.289970 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jp78g"] Oct 31 02:55:15.510721 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.510678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:15.510913 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.510806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:15.510913 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:15.510852 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:15.511012 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:15.510921 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 02:55:15.511012 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:15.510953 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:19.510928212 +0000 UTC m=+38.182830553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:15.511012 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:15.510980 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs podName:22743d27-b50f-4219-a8b7-198e417780ee nodeName:}" failed. No retries permitted until 2026-10-31 02:55:47.510963462 +0000 UTC m=+66.182865788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs") pod "network-metrics-daemon-29cdk" (UID: "22743d27-b50f-4219-a8b7-198e417780ee") : secret "metrics-daemon-secret" not found Oct 31 02:55:15.611460 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.611407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:15.611642 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.611483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:15.614566 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.614535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:15.615966 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.615944 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:15.828965 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.828890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:15.839060 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:15.839022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:16.022740 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.022697 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" event={"ID":"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36","Type":"ContainerStarted","Data":"848f1416c4c96e769a374c989510a0edda0ea5c324ee5f3c5226f792cb3d9282"} Oct 31 02:55:16.024791 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.024759 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" event={"ID":"5248cdbd-3ee6-43c8-826b-0314c2c7644e","Type":"ContainerStarted","Data":"ec79bd35e285e37940232b9109f89986d1b5d23c5067dc16e0b6e71d5346a277"} Oct 31 02:55:16.046375 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.046312 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" podStartSLOduration=2.321257504 podStartE2EDuration="4.04629374s" totalImagesPullingTime="1.725036236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:16.044176542 +0000 UTC m=+34.716078912" watchObservedRunningTime="2026-10-31 02:55:16.04629374 +0000 UTC m=+34.718196083" Oct 31 02:55:16.049716 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.049691 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:16.113795 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.113716 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5q2kq"] Oct 31 02:55:16.140596 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.140409 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8dxn"] Oct 31 02:55:16.217186 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.217127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:16.253447 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:55:16.250941 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636e54a6_ea98_41b1_9f02_d11a89ad344f.slice/crio-92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee WatchSource:0}: Error finding container 92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee: Status 404 returned error can't find the container with id 92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee Oct 31 02:55:16.253447 ip-10-0-139-52 kubenswrapper[2583]: W1031 02:55:16.251253 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bf57c6_bc1c_4f99_9e94_5f36c30f4bec.slice/crio-a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d WatchSource:0}: Error finding container a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d: Status 404 returned error can't find the container with id a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d Oct 31 02:55:16.520154 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.520116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:16.520318 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.520170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:16.520318 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:16.520280 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:16.520469 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:16.520348 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:20.520331502 +0000 UTC m=+39.192233829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:16.520752 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:16.520731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:17.029395 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.029354 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7ngz" event={"ID":"191190f6-9b6d-498d-b960-67437ac0b983","Type":"ContainerStarted","Data":"1e9ac2219098a45452e31ad2607b36f6b265af4fec1b5a789ab47eed556ce075"} Oct 31 02:55:17.030896 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.030864 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8dxn" event={"ID":"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec","Type":"ContainerStarted","Data":"a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d"} Oct 31 02:55:17.032603 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.032577 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" event={"ID":"503a089d-2343-450d-bba6-486321ca52e4","Type":"ContainerStarted","Data":"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7"} Oct 31 02:55:17.034991 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.034967 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5q2kq" event={"ID":"636e54a6-ea98-41b1-9f02-d11a89ad344f","Type":"ContainerStarted","Data":"92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee"} Oct 31 02:55:17.052005 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.051944 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7ngz" podStartSLOduration=2.15895686 podStartE2EDuration="6.05192607s" totalImagesPullingTime="3.89296921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:17.051196295 +0000 UTC m=+35.723098661" watchObservedRunningTime="2026-10-31 02:55:17.05192607 +0000 UTC m=+35.723828414" Oct 31 02:55:17.075555 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:17.075496 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" podStartSLOduration=2.106463485 podStartE2EDuration="5.075474953s" totalImagesPullingTime="2.969011468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:17.074620539 +0000 UTC m=+35.746522881" watchObservedRunningTime="2026-10-31 02:55:17.075474953 +0000 UTC m=+35.747377296" Oct 31 02:55:19.034124 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.034082 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:19.041649 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.041622 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" event={"ID":"5248cdbd-3ee6-43c8-826b-0314c2c7644e","Type":"ContainerStarted","Data":"4a943aa73d160012fe08119801c4c04e51d33da0b152d4013f6876ca7f7ca3fc"} Oct 31 02:55:19.061551 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.061482 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" podStartSLOduration=1.5448377880000002 podStartE2EDuration="5.06146458s" totalImagesPullingTime="3.516626792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:19.060091459 +0000 UTC m=+37.731993793" watchObservedRunningTime="2026-10-31 02:55:19.06146458 +0000 UTC m=+37.733366921" Oct 31 02:55:19.544354 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.544308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:19.544566 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:19.544473 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:19.544566 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:19.544537 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:27.544518781 +0000 UTC m=+46.216421103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:19.858960 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.858878 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:19.987000 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:19.986858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:20.248995 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:20.248942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:20.385209 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:20.385170 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:20.561075 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:20.560985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:20.561239 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:20.561167 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:20.561239 ip-10-0-139-52 kubenswrapper[2583]: E1031 02:55:20.561238 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:28.561220123 +0000 UTC m=+47.233122458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:21.007177 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.007113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:21.049736 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.049697 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5q2kq" event={"ID":"636e54a6-ea98-41b1-9f02-d11a89ad344f","Type":"ContainerStarted","Data":"23124ad65c0c17526469cfd8d944561e6b643be8751d99336662cccfcfa36ae8"} Oct 31 02:55:21.049920 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.049826 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:21.068970 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.068876 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5q2kq" podStartSLOduration=35.616704476 podStartE2EDuration="40.068864159s" totalImagesPullingTime="4.452159683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:21.068405737 +0000 UTC m=+39.740308081" watchObservedRunningTime="2026-10-31 02:55:21.068864159 +0000 UTC m=+39.740766529" Oct 31 02:55:21.614919 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.614885 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:21.810805 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:21.810774 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:22.006334 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:22.006297 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:22.053694 ip-10-0-139-52 kubenswrapper[2583]: I1031 02:55:22.053661 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8dxn" event={"ID":"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec","Type":"ContainerStarted","Data":"a1d8f8d8fa1edaf1faeceb226207c825cf46a5d233987641fcce5f5ed77fcdb8"} Oct 31 02:55:22.057360 ip-10-0-139-52 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 02:55:22.073842 ip-10-0-139-52 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 02:55:22.074016 ip-10-0-139-52 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 02:55:22.074070 ip-10-0-139-52 systemd[1]: kubelet.service: Consumed 2.501s CPU time. Oct 31 02:55:22.085230 ip-10-0-139-52 systemd[1]: Starting Kubernetes Kubelet... Oct 31 02:55:22.175793 ip-10-0-139-52 kubenswrapper[4556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:22.175793 ip-10-0-139-52 kubenswrapper[4556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 02:55:22.175793 ip-10-0-139-52 kubenswrapper[4556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:22.175793 ip-10-0-139-52 kubenswrapper[4556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 02:55:22.177854 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177774 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.177854 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177853 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.177854 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177857 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177860 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177863 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177866 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177869 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177872 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177880 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177883 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177886 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177889 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177892 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177895 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177898 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177900 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177903 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177906 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177908 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177911 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177913 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177916 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.177949 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177919 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177922 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177925 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177927 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177930 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177933 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177936 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177939 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177942 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177945 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177948 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177951 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177954 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177957 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177960 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177962 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177965 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177968 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177971 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.178476 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177974 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177976 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177979 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177982 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177985 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177987 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177990 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177994 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.177996 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178000 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178005 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178008 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178011 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178013 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178016 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178020 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178025 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178028 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.178942 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178031 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178034 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178036 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178039 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178042 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178045 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178048 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178050 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178053 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178056 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178058 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178061 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178064 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178067 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178070 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178073 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178075 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178078 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178080 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178083 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.179382 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178085 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178088 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178091 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178094 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178096 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178099 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178101 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178104 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178107 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178110 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178113 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178116 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178118 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178121 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178124 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178126 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178132 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178135 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178138 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178141 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.179882 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178143 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178146 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178149 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178151 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178154 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178157 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178172 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178930 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178937 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178940 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178943 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178946 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178949 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178951 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178954 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178958 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178961 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178964 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178967 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178969 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.180366 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178973 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178978 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178981 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178984 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178987 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178989 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178992 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.178995 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179000 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179003 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179006 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179008 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179011 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179014 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179017 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179019 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179022 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179024 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179027 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.180872 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179035 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179038 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179041 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179044 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179047 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179049 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179052 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179054 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179057 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179060 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179062 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179065 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179067 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179070 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179073 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179076 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179078 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179081 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179084 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179087 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.181349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179089 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179094 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179097 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179100 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179103 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179105 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179108 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179110 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179113 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179116 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179118 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179122 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179124 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179127 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179130 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179133 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179136 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179141 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179144 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.181856 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179147 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179150 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179153 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179156 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179158 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179161 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179164 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179167 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179169 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179172 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179175 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179177 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179180 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179182 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179190 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179192 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179195 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179198 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179201 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.182331 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179203 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179206 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179209 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179211 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179214 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179216 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179219 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179222 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179225 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179228 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179230 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179234 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179237 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179241 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179243 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179246 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179740 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179746 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179750 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.182861 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179753 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179756 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179758 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179762 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179764 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179767 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179770 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179772 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179776 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179779 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179781 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179784 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179787 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179789 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179792 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179794 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179797 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179799 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179802 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.183327 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179804 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179807 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179810 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179813 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179816 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179819 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179822 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179824 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179827 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179830 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179832 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179835 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179838 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179840 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179843 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179847 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179851 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179854 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179857 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179859 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.183827 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179862 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179865 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179867 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179870 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179872 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179875 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179877 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179880 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179883 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179886 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179888 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179891 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179894 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179896 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179899 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179904 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179906 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179909 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179912 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.184301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179914 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179917 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179919 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179922 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179925 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179927 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179930 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179932 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179935 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179937 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179940 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179942 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179945 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179947 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179950 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179953 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179956 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179958 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179961 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179963 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.184768 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179966 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179968 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179971 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179973 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179977 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179979 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179982 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179985 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179989 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179992 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179995 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.179998 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180000 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180003 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180005 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180008 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180010 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180013 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180015 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.185553 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180018 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180020 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180023 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180026 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180028 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180031 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180123 4556 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180130 4556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180137 4556 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180143 4556 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180147 4556 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180151 4556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180156 4556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180160 4556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180163 4556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180167 4556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180171 4556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180174 4556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180177 4556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180180 4556 flags.go:64] FLAG: --cgroup-root="" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180184 4556 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180187 4556 flags.go:64] FLAG: --client-ca-file="" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180190 4556 flags.go:64] FLAG: --cloud-provider="external" Oct 31 02:55:22.186211 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180193 4556 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180198 4556 flags.go:64] FLAG: --cluster-domain="" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180200 4556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180203 4556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180207 4556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180210 4556 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180214 4556 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180217 4556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180220 4556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180223 4556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180226 4556 flags.go:64] FLAG: --contention-profiling="false" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180232 4556 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180235 4556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180238 4556 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180241 4556 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180245 4556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180248 4556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180251 4556 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180254 4556 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180257 4556 flags.go:64] FLAG: --enable-server="true" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180262 4556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180267 4556 flags.go:64] FLAG: --event-burst="100" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180270 4556 flags.go:64] FLAG: --event-qps="50" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180273 4556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 02:55:22.186845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180277 4556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180280 4556 flags.go:64] FLAG: --eviction-hard="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180284 4556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180287 4556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180290 4556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180293 4556 flags.go:64] FLAG: --eviction-soft="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180296 4556 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180299 4556 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180302 4556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180305 4556 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180309 4556 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180312 4556 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180314 4556 flags.go:64] FLAG: --feature-gates="" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180321 4556 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180324 4556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180328 4556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180331 4556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180334 4556 flags.go:64] FLAG: --healthz-port="10248" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180338 4556 flags.go:64] FLAG: --help="false" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180342 4556 flags.go:64] FLAG: --hostname-override="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180346 4556 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180349 4556 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180352 4556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 02:55:22.187459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180356 4556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180359 4556 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180363 4556 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180366 4556 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180369 4556 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180373 4556 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180376 4556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180380 4556 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180383 4556 flags.go:64] FLAG: --kube-reserved="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180386 4556 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180389 4556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180392 4556 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180395 4556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180398 4556 flags.go:64] FLAG: --lock-file="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180400 4556 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180404 4556 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180407 4556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180425 4556 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180428 4556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180432 4556 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180435 4556 flags.go:64] FLAG: --logging-format="text" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180438 4556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180442 4556 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180445 4556 flags.go:64] FLAG: --manifest-url="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180447 4556 flags.go:64] FLAG: --manifest-url-header="" Oct 31 02:55:22.188050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180452 4556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180455 4556 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180459 4556 flags.go:64] FLAG: --max-pods="110" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180463 4556 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180466 4556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180469 4556 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180473 4556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180476 4556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180480 4556 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180483 4556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180490 4556 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180493 4556 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180497 4556 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180500 4556 flags.go:64] FLAG: --pod-cidr="" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180503 4556 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180506 4556 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180510 4556 flags.go:64] FLAG: --pods-per-core="0" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180512 4556 flags.go:64] FLAG: --port="10250" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180515 4556 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180518 4556 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-075d65ea86d0e6a73" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180522 4556 flags.go:64] FLAG: --qos-reserved="" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180525 4556 flags.go:64] FLAG: --read-only-port="10255" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180528 4556 flags.go:64] FLAG: --register-node="true" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180531 4556 flags.go:64] FLAG: --register-with-taints="" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180534 4556 flags.go:64] FLAG: --registry-burst="10" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180537 4556 flags.go:64] FLAG: --registry-qps="5" Oct 31 02:55:22.188683 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180540 4556 flags.go:64] FLAG: --reserved-cpus="" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180544 4556 flags.go:64] FLAG: --reserved-memory="" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180548 4556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180551 4556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180554 4556 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180557 4556 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180560 4556 flags.go:64] FLAG: --runonce="false" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180563 4556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180566 4556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180570 4556 flags.go:64] FLAG: --seccomp-default="false" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180573 4556 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180576 4556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180580 4556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180583 4556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180586 4556 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180589 4556 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180592 4556 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180595 4556 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180601 4556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180604 4556 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180607 4556 flags.go:64] FLAG: --system-cgroups="" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180610 4556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180616 4556 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180618 4556 flags.go:64] FLAG: --tls-cert-file="" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180622 4556 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 02:55:22.189327 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180626 4556 flags.go:64] FLAG: --tls-min-version="" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180629 4556 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180632 4556 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180635 4556 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180638 4556 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180641 4556 flags.go:64] FLAG: --v="2" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180646 4556 flags.go:64] FLAG: --version="false" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180650 4556 flags.go:64] FLAG: --vmodule="" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180654 4556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.180658 4556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180794 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180798 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180801 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180804 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180806 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180809 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180813 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180816 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180818 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180821 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180824 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.189934 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180827 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180829 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180832 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180834 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180839 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180841 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180844 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180847 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180849 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180852 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180855 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180858 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180861 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180863 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180868 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180872 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180875 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180878 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180881 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.190560 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180885 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180889 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180892 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180895 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180898 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180901 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180903 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180906 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180912 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180914 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180917 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180919 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180922 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180925 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180927 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180930 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180932 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180936 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180939 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.191155 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180942 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180945 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180947 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180950 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180952 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180955 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180957 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180960 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180962 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180965 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180968 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180970 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180973 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180976 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180978 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180981 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180984 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180987 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180989 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180992 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.191774 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180994 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.180999 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181001 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181004 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181006 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181009 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181012 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181014 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181017 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181019 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181023 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181026 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181029 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181032 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181034 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181037 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181040 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181042 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181045 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181048 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.192301 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181050 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181053 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181055 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181058 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181060 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181063 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181065 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181068 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181071 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181074 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181077 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181079 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181082 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181086 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181089 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181091 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.192821 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181094 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.181119 4556 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181154 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181158 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181160 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181163 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181166 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181170 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181173 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181176 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181179 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181182 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181184 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181187 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181189 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181192 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.193266 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181195 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181197 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181200 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181203 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181205 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181208 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181211 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181213 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181216 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181219 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181222 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181225 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181227 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181231 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181234 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181237 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181239 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181242 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181244 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.193910 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181247 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181250 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181253 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181256 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181260 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181262 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181265 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181268 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181270 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181273 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181276 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181278 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181281 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181284 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181286 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181289 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181291 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181296 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181300 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.194438 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181303 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181306 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181309 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181314 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181317 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181321 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181324 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181328 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181331 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181334 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181336 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181339 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181341 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181344 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181347 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181349 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181352 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181356 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181359 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.194900 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181362 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181365 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181367 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181370 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181372 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181375 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181378 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181380 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181383 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181385 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181388 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181391 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181394 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181396 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181399 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181402 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181404 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181407 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181423 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.195445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181426 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181430 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181445 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181448 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181450 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181453 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181456 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181458 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181461 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181464 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181466 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181471 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181473 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181476 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181480 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.181482 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.185468 4556 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.185490 4556 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.185514 4556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.185522 4556 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 02:55:22.195927 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185686 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185693 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185699 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185703 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185708 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185713 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185717 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185721 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185726 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185731 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185735 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185740 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185749 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185754 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185758 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185763 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185767 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185771 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185776 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185780 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.196454 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185786 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185790 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185794 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185799 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185803 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185812 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185817 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185822 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185826 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185832 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185836 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185841 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185845 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185849 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185854 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185858 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185863 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185868 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185877 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.196930 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185882 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185887 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185891 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185896 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185901 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185905 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185910 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185914 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185919 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185924 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185928 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185933 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185942 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185946 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185950 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185955 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185959 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185964 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185968 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.185992 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.197445 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186000 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186006 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186011 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186016 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186021 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186027 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186074 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186093 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186192 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186197 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186200 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186203 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186207 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186210 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186213 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186216 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186220 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186223 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186226 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.197946 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186228 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186231 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186235 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186237 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186240 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186243 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186246 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186249 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186251 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186256 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186260 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186264 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186268 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186271 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186274 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186277 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186279 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186283 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186286 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.198397 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186288 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186291 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186294 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186297 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186299 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186302 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186304 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186307 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186309 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.186346 4556 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186521 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186528 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186532 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186535 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186538 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.198903 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186541 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186543 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186546 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186549 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186552 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186555 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186557 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186560 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186562 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186565 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186568 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186570 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186573 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186576 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186579 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186581 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186584 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186587 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186589 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.199274 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186592 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186595 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186598 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186600 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186603 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186605 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186608 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186612 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186617 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186619 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186622 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186625 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186628 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186630 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186634 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186637 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186639 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186642 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186644 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186647 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.199864 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186649 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186652 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186654 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186657 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186659 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186662 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186665 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186667 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186669 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186672 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186675 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186677 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186680 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186685 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186688 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186691 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186694 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186697 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186699 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186702 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.200379 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186705 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186707 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186710 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186713 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186716 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186720 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186723 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186727 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186730 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186732 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186735 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186738 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186740 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186743 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186746 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186748 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186751 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186754 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186757 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.200876 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186759 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186762 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186765 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186767 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186770 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186773 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186777 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186780 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186783 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186785 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186788 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186791 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186793 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186797 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186799 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186802 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186805 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186807 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186810 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186812 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.201349 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186815 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186818 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186821 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.186846 4556 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186877 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186882 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186885 4556 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186888 4556 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186891 4556 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186894 4556 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186896 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186899 4556 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186901 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186904 4556 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186906 4556 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186909 4556 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 02:55:22.202016 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186912 4556 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186914 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186917 4556 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186921 4556 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186924 4556 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186926 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186929 4556 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186931 4556 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186934 4556 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186937 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186940 4556 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186943 4556 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186945 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186948 4556 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186951 4556 feature_gate.go:386] unrecognized feature gate: Example Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186953 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186956 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186959 4556 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186962 4556 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186965 4556 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 02:55:22.202656 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186967 4556 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186970 4556 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186973 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186975 4556 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186980 4556 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186983 4556 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186986 4556 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186989 4556 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186991 4556 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186994 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186997 4556 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.186999 4556 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187002 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187005 4556 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187007 4556 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187012 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187015 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187017 4556 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187020 4556 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 02:55:22.203262 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187023 4556 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187026 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187029 4556 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187032 4556 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187034 4556 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187037 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187040 4556 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187043 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187045 4556 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187048 4556 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187050 4556 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187053 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187057 4556 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187059 4556 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187062 4556 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187064 4556 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187067 4556 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187070 4556 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187072 4556 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187075 4556 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 02:55:22.203795 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187077 4556 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187080 4556 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187083 4556 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187085 4556 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187088 4556 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187091 4556 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187094 4556 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187096 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187100 4556 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187103 4556 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187105 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187108 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187110 4556 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187113 4556 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187116 4556 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187118 4556 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187121 4556 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187124 4556 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187126 4556 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187129 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 02:55:22.204299 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187131 4556 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187134 4556 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187136 4556 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187139 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187142 4556 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187145 4556 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187148 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187150 4556 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187153 4556 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187155 4556 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187158 4556 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187160 4556 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187163 4556 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187167 4556 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:22.187171 4556 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.187225 4556 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188256 4556 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188312 4556 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 02:55:22.204937 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188503 4556 server.go:1014] "Starting client certificate rotation" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188637 4556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188693 4556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 02:49:41 +0000 UTC" deadline="2028-07-08 20:56:08.896969302 +0000 UTC" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.188735 4556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14802h0m46.708247296s" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.189002 4556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.190030 4556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.191605 4556 log.go:25] "Validated CRI v1 runtime API" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.193177 4556 log.go:25] "Validated CRI v1 image API" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.193989 4556 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 02:55:22.205650 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.195507 4556 fs.go:135] Filesystem UUIDs: map[550a4c21-a443-40a6-9537-b32bc2259b88:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fcff9b2c-71a1-4958-bedb-3fce9b2743f2:/dev/nvme0n1p4] Oct 31 02:55:22.206084 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.195527 4556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec79bd35e285e37940232b9109f89986d1b5d23c5067dc16e0b6e71d5346a277/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec79bd35e285e37940232b9109f89986d1b5d23c5067dc16e0b6e71d5346a277/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c89547-d1bb-4dfa-94e5-cd303ed05f2a/volumes/kubernetes.io~projected/kube-api-access-pm24j:{mountpoint:/var/lib/kubelet/pods/12c89547-d1bb-4dfa-94e5-cd303ed05f2a/volumes/kubernetes.io~projected/kube-api-access-pm24j major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~projected/kube-api-access-bjqzc:{mountpoint:/var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~projected/kube-api-access-bjqzc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~projected/kube-api-access-5ljwr:{mountpoint:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~projected/kube-api-access-5ljwr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22743d27-b50f-4219-a8b7-198e417780ee/volumes/kubernetes.io~projected/kube-api-access-4csml:{mountpoint:/var/lib/kubelet/pods/22743d27-b50f-4219-a8b7-198e417780ee/volumes/kubernetes.io~projected/kube-api-access-4csml major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/371390b5-879d-403c-b7e5-58bbceb94ba0/volumes/kubernetes.io~projected/kube-api-access-4n8xg:{mountpoint:/var/lib/kubelet/pods/371390b5-879d-403c-b7e5-58bbceb94ba0/volumes/kubernetes.io~projected/kube-api-access-4n8xg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36/volumes/kubernetes.io~projected/kube-api-access-dlb8h:{mountpoint:/var/lib/kubelet/pods/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36/volumes/kubernetes.io~projected/kube-api-access-dlb8h major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/503a089d-2343-450d-bba6-486321ca52e4/volumes/kubernetes.io~projected/kube-api-access-d8zzx:{mountpoint:/var/lib/kubelet/pods/503a089d-2343-450d-bba6-486321ca52e4/volumes/kubernetes.io~projected/kube-api-access-d8zzx major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~projected/kube-api-access-tkd9q:{mountpoint:/var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~projected/kube-api-access-tkd9q major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~secret/signing-key major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/636e54a6-ea98-41b1-9f02-d11a89ad344f/volumes/kubernetes.io~projected/kube-api-access-gq42q:{mountpoint:/var/lib/kubelet/pods/636e54a6-ea98-41b1-9f02-d11a89ad344f/volumes/kubernetes.io~projected/kube-api-access-gq42q major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8246785f-965d-4107-9e6e-1bed982b3d0e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8246785f-965d-4107-9e6e-1bed982b3d0e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/833330df-2c89-48b4-9c1d-c049b3378e26/volumes/kubernetes.io~projected/kube-api-access-dxvkt:{mountpoint:/var/lib/kubelet/pods/833330df-2c89-48b4-9c1d-c049b3378e26/volumes/kubernetes.io~projected/kube-api-access-dxvkt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aef181b7-b312-4f41-b898-be29316d5638/volumes/kubernetes.io~projected/kube-api-access-j2tzx:{mountpoint:/var/lib/kubelet/pods/aef181b7-b312-4f41-b898-be29316d5638/volumes/kubernetes.io~projected/kube-api-access-j2tzx major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d8733a-ce73-455e-aeb6-bba45d34b97f/volumes/kubernetes.io~projected/kube-api-access-w2lzn:{mountpoint:/var/lib/kubelet/pods/c1d8733a-ce73-455e-aeb6-bba45d34b97f/volumes/kubernetes.io~projected/kube-api-access-w2lzn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb3abcfe-0522-4dfb-abda-7f0707b65a65/volumes/kubernetes.io~projected/kube-api-access-5wbv4:{mountpoint:/var/lib/kubelet/pods/cb3abcfe-0522-4dfb-abda-7f0707b65a65/volumes/kubernetes.io~projected/kube-api-access-5wbv4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd57deac-c763-4eb7-9010-8d38ea97766c/volumes/kubernetes.io~projected/kube-api-access-2kb8c:{mountpoint:/var/lib/kubelet/pods/dd57deac-c763-4eb7-9010-8d38ea97766c/volumes/kubernetes.io~projected/kube-api-access-2kb8c major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4/volumes/kubernetes.io~projected/kube-api-access-pswzg:{mountpoint:/var/lib/kubelet/pods/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4/volumes/kubernetes.io~projected/kube-api-access-pswzg major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~projected/kube-api-access-kt6ln:{mountpoint:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~projected/kube-api-access-kt6ln major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/496b7e5fa251531f4a8a5377e6a8c00f7c32e7107025be64a78b6e361a293ee4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/673b60a25a6b2a38d0d0fd1563edd4357260ee4b0ccf607a2c6f91a7b8125a23/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1b16bc30346f5903e57975b5b50346ee68bd917553ab9e96ffc5ac6907ea486e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/12670b83ab5e91410c957666308b410d3e35f2b411cb120682ddce4a5158a291/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/738a90925bc5959598fc52f5990144b6afccfde26a01aea9ec08102ce134db94/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a3ffa203c2524066959e1992cf13a7b5be056744ea454c55d6872db72ac5f772/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b9ebf5763c32896d1fed5d6e71f90a26e374bd561a2dc298717fb2d20f0dafd5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/25a4a13cb9f306b0f7dfef3e2398db8691bee13643bcd9610771aea2d9b8a36e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/4596905c3bfca2a845ed84fb2619efe5de41c62f5cce4e6b4005dec05a9ae667/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/88603931d60d78566b40a261d8b4a293596621749e4e9f3e7203112f7e425a45/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/0a4ac08095b2f77b955d8495a695a899bb05e866c59d739bd1727555921ca619/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/536239eb7069003ad3ec2b34000d0ce591f3d05797d5974f442b8d5a29ade258/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3ff7c84b59e09d33353d513913bb37cf0daa6f21cdd0a411e72eb5e23d37126b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cd57535d1f1ec674f05cf2d176648d7df873484a2ed95a0d97df2eda3108cc51/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7d706e40dac04dfbcc1a3b475a650d3ebcff0c15ed39ef07500526da992e0a9e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9e4a0715287df2e4b2cab54db134dfc944d567e6fab1c69c9b0682b21df2ae46/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dda9981fa1e36bb3170ac9d39df22add71232b4a71a7f28039f7a70388911923/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/98da48326d460307006c062a84c50aa36b7c47424c2721e976f30e3009ee492d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a54aca8e6147fa89352d68c8e54bca6f89fd556efd739585cfb1f99bf2619ff6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/82d2b71befc44cab7035c3836ecf7dc34293c774b3c1c7a43a5f961418e47fd4/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4a8eaf1525dc2cd7c58cbd0c4caedd17920dccdf857565783208df50708f50ea/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/2e36e0d922600b6694c579196cb126f655b3e026623e2382081a0eae92485333/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ca69b9f1270e1d0392c7852b26946574be3466c732e9bdf03340f3e329e99a8b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2847299e7fdf2952cd2cb087a2eb469e7c90e0a7f23ff0167666340ec70aee25/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/44687d1ef4d129347816d4b8b6368b4c79b9220963eb811c469fd8b0bbd6f5ad/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/9b648e63f6395bea9dae971d794157fbfa6a7e450c39ba3b06209b9d991dc071/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/aeec181623dab6da8a98703e2853d4a8e188e00347128b68d28c23757c7f6bfc/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/f8982ed84fc7b5610351f82f35dec19adeceb29af93c67da4c89d9977e6a2a7b/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/2ef2c4466485a972a85545568fc1b2305e71af9cf8ffb7a1a403e1f7bc49a648/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/c5ab76b846c3e67726403b9e35e2d80f904594a6ad1b39cebaf06d1e960bb37d/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/46cb9986cf950d4ce4344c89a6b8a887a1d66641a74bbc2f2a3f8b2a485dfb70/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/f8b30f6c2b1fa95ecf5b5868c0faabba31e9d72a0d454080041f6e3e55842e02/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/4db433365c6e7c9e8c5c62450ea1106697f5095856e3c231136a16da69341506/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7f8481a20986383336f88af33c5bf30d62b18605878a320fec3e753d8fa84dbf/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/12722186b55702bcaecc277c3b3b7a9feff61adbfecd05bc7946e00be89287c2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4a3ea81e6bd38be638d0a2c2e02992c1cc15a0f90e50526af386791567c2d885/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/765aa572f0174f248450e984aaedac11e93f79c75920026c3b92967308f3ff46/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e99a5738828012c67d76f4d4600c0e93a174f2547c008d79c5abbdd269d66b02/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/76170af74a199db3337c04360e4a5e5fa38016495bca67263909fd790ad9d7cc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f171f5f8ef9877c5bbc03e51dd037dbc8b601fddf1d4dd1e7ac92369de35495d/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/37e498a55b7234df93720d18c1f9983d0b269694d8fff6eb1cc4e3fffbd31729/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/069e793b73964ddf6b1e39afba7f7c18995f7ed603bdc9e9111fad001491d01c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4e7044a7a3edc6e983b7f7c4e6fabe20d9f845dbc5af6d0870b637219970a6af/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.202579 4556 manager.go:217] Machine: {Timestamp:2026-10-31 02:55:22.201770418 +0000 UTC m=+0.056787551 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108581 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22e3f7be11e4f6a83ac63ea20a5981 SystemUUID:ec22e3f7-be11-e4f6-a83a-c63ea20a5981 BootID:6ba96047-ef76-4e7f-9494-236e190692f7 Filesystems:[{Device:/run/containers/storage/overlay-containers/a8d72714d86b51ade85b6a242810256bd1ce775e7995ce149eec52d6dac3b57d/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4/volumes/kubernetes.io~projected/kube-api-access-pswzg DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/636e54a6-ea98-41b1-9f02-d11a89ad344f/volumes/kubernetes.io~projected/kube-api-access-gq42q DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92827c1ae0f871e2f5bd33bda1bf35c9011d66c732453ab4fd753364d097d8ee/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d8733a-ce73-455e-aeb6-bba45d34b97f/volumes/kubernetes.io~projected/kube-api-access-w2lzn DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8246785f-965d-4107-9e6e-1bed982b3d0e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12c89547-d1bb-4dfa-94e5-cd303ed05f2a/volumes/kubernetes.io~projected/kube-api-access-pm24j DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22743d27-b50f-4219-a8b7-198e417780ee/volumes/kubernetes.io~projected/kube-api-access-4csml DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~projected/kube-api-access-5ljwr DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/371390b5-879d-403c-b7e5-58bbceb94ba0/volumes/kubernetes.io~projected/kube-api-access-4n8xg DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5909d314269100895824c49157b8ccbedaa2a5337c59ed594d56fb7033973d59/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~projected/kube-api-access-bjqzc DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5c45319876b38a80b73f99690091fb607cb914e7972e9c420d16fbca8687c1e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/503a089d-2343-450d-bba6-486321ca52e4/volumes/kubernetes.io~projected/kube-api-access-d8zzx DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36/volumes/kubernetes.io~projected/kube-api-access-dlb8h DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd57deac-c763-4eb7-9010-8d38ea97766c/volumes/kubernetes.io~projected/kube-api-access-2kb8c DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/833330df-2c89-48b4-9c1d-c049b3378e26/volumes/kubernetes.io~projected/kube-api-access-dxvkt DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b18bbdff3c02fbdb59d5cba842a2b484cf15ba8a61a8323decafbadeadb8b0e9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ceb12c9adb8fa5e0c108829653ee3279883353a4e5ddd81314c586f4718aa582/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b883ea7ed532bcc2ab6fa46d8604aae67d83bf1ef8bfe6b9027e42666481539e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/191190f6-9b6d-498d-b960-67437ac0b983/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e4d369a854facfe0ff4522ac7a0e3e08b28ef0712f77d773840c635ba5fa6d5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec79bd35e285e37940232b9109f89986d1b5d23c5067dc16e0b6e71d5346a277/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe728edf-d50f-4c7b-b98c-a12c19c39fa6/volumes/kubernetes.io~projected/kube-api-access-kt6ln DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8220a2508b4c85fdb3ff93a62141389e13bcf6bae027b793da7b2dcff95fc09/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb3abcfe-0522-4dfb-abda-7f0707b65a65/volumes/kubernetes.io~projected/kube-api-access-5wbv4 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5248cdbd-3ee6-43c8-826b-0314c2c7644e/volumes/kubernetes.io~projected/kube-api-access-tkd9q DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/026733f062c1156fc57189bcff1cb86873a09262e4ada334bd97a8f0c124ee66/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42a18e16cfe5fb243ceca3d017620ee84c06f2d3547ee7c7518cdf6dfb2b054a/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2e195d6a35005b3171c3c4ba616a4f0b70743b0a7e46d363bce487004f620d4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/aef181b7-b312-4f41-b898-be29316d5638/volumes/kubernetes.io~projected/kube-api-access-j2tzx DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:42a18e16cfe5fb2 MacAddress:72:dc:94:1c:c7:11 Speed:10000 Mtu:8901} {Name:75c04598e48138c MacAddress:fa:4d:0f:ee:09:29 Speed:10000 Mtu:8901} {Name:92827c1ae0f871e MacAddress:6a:7c:a4:c4:ee:eb Speed:10000 Mtu:8901} {Name:a8d72714d86b51a MacAddress:96:95:35:bb:5d:35 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dc:7a:5f:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:86:3e:de:41:80:2d Speed:0 Mtu:8901} {Name:c8220a2508b4c85 MacAddress:6a:d7:17:b3:41:b7 Speed:10000 Mtu:8901} {Name:ec79bd35e285e37 MacAddress:4a:a9:16:e2:34:09 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dc:7a:5f:9f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:f0:3a:b1:af:ff Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:a1:65:7a:b6:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203305 4556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203359 4556 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203648 4556 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203667 4556 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203798 4556 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203809 4556 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203827 4556 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203847 4556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.203942 4556 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204123 4556 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204248 4556 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204285 4556 kubelet.go:493] "Attempting to sync node with API server" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204298 4556 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204307 4556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204316 4556 kubelet.go:405] "Adding apiserver pod source" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204323 4556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204885 4556 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.204895 4556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:22.206339 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.205359 4556 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 02:55:22.208083 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208064 4556 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 02:55:22.208083 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208080 4556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 02:55:22.208220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208207 4556 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 02:55:22.208451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208435 4556 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 02:55:22.208527 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208476 4556 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 02:55:22.208602 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208591 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 02:55:22.208661 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208611 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 02:55:22.208661 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208622 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 02:55:22.208661 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208631 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 02:55:22.208661 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208641 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 02:55:22.208661 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208660 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208669 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208680 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208691 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208700 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208720 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208735 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208770 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 02:55:22.208883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.208780 4556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 02:55:22.212130 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.212110 4556 server.go:1263] "Started kubelet" Oct 31 02:55:22.212866 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.212200 4556 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 02:55:22.212866 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.212206 4556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 02:55:22.212866 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.212401 4556 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 02:55:22.212866 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.212870 4556 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 02:55:22.212669 ip-10-0-139-52 systemd[1]: Started Kubernetes Kubelet. Oct 31 02:55:22.214080 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.214050 4556 server.go:317] "Adding debug handlers to kubelet server" Oct 31 02:55:22.215320 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.215302 4556 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.215748 4556 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.215765 4556 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.215876 4556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.215885 4556 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-52.us-east-2.compute.internal\" not found" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216126 4556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216170 4556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 02:49:43 +0000 UTC" deadline="2028-07-18 23:27:47.23295257 +0000 UTC" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216200 4556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15044h32m25.016755911s" Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216661 4556 factory.go:153] Registering CRI-O factory Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216678 4556 factory.go:223] Registration of the crio container factory successfully Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216722 4556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216729 4556 factory.go:55] Registering systemd factory Oct 31 02:55:22.216740 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216736 4556 factory.go:223] Registration of the systemd container factory successfully Oct 31 02:55:22.217347 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216756 4556 factory.go:103] Registering Raw factory Oct 31 02:55:22.217347 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.216766 4556 manager.go:1199] Started watching for new ooms in manager Oct 31 02:55:22.217347 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.217044 4556 manager.go:319] Starting recovery of all containers Oct 31 02:55:22.219318 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219275 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7" volumeName="kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides" seLinuxMountContext="" Oct 31 02:55:22.219403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219334 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833330df-2c89-48b4-9c1d-c049b3378e26" volumeName="kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config" seLinuxMountContext="" Oct 31 02:55:22.219403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219352 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe728edf-d50f-4c7b-b98c-a12c19c39fa6" volumeName="kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp" seLinuxMountContext="" Oct 31 02:55:22.219403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219369 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22743d27-b50f-4219-a8b7-198e417780ee" volumeName="kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml" seLinuxMountContext="" Oct 31 02:55:22.219403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219391 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503a089d-2343-450d-bba6-486321ca52e4" volumeName="kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle" seLinuxMountContext="" Oct 31 02:55:22.219608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219405 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef181b7-b312-4f41-b898-be29316d5638" volumeName="kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 02:55:22.219608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219441 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3abcfe-0522-4dfb-abda-7f0707b65a65" volumeName="kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:22.219608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219520 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3abcfe-0522-4dfb-abda-7f0707b65a65" volumeName="kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 02:55:22.219608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219545 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea5034e0-53b0-48d6-8d9e-71c8eb577ca4" volumeName="kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg" seLinuxMountContext="" Oct 31 02:55:22.219608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219588 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7" volumeName="kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219601 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503a089d-2343-450d-bba6-486321ca52e4" volumeName="kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219649 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7" volumeName="kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219661 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7" volumeName="kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219680 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503a089d-2343-450d-bba6-486321ca52e4" volumeName="kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219696 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5248cdbd-3ee6-43c8-826b-0314c2c7644e" volumeName="kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219716 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5248cdbd-3ee6-43c8-826b-0314c2c7644e" volumeName="kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219736 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8246785f-965d-4107-9e6e-1bed982b3d0e" volumeName="kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219748 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3abcfe-0522-4dfb-abda-7f0707b65a65" volumeName="kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 02:55:22.219836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219764 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb3abcfe-0522-4dfb-abda-7f0707b65a65" volumeName="kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219846 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="191190f6-9b6d-498d-b960-67437ac0b983" volumeName="kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219865 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="636e54a6-ea98-41b1-9f02-d11a89ad344f" volumeName="kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219878 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833330df-2c89-48b4-9c1d-c049b3378e26" volumeName="kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219890 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef181b7-b312-4f41-b898-be29316d5638" volumeName="kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219906 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aef181b7-b312-4f41-b898-be29316d5638" volumeName="kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219919 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd57deac-c763-4eb7-9010-8d38ea97766c" volumeName="kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219937 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd57deac-c763-4eb7-9010-8d38ea97766c" volumeName="kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.219951 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea5034e0-53b0-48d6-8d9e-71c8eb577ca4" volumeName="kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220009 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c89547-d1bb-4dfa-94e5-cd303ed05f2a" volumeName="kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220034 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="191190f6-9b6d-498d-b960-67437ac0b983" volumeName="kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220053 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="371390b5-879d-403c-b7e5-58bbceb94ba0" volumeName="kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220065 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503a089d-2343-450d-bba6-486321ca52e4" volumeName="kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220082 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5248cdbd-3ee6-43c8-826b-0314c2c7644e" volumeName="kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220093 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d8733a-ce73-455e-aeb6-bba45d34b97f" volumeName="kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220146 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea5034e0-53b0-48d6-8d9e-71c8eb577ca4" volumeName="kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220163 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe728edf-d50f-4c7b-b98c-a12c19c39fa6" volumeName="kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220173 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c89547-d1bb-4dfa-94e5-cd303ed05f2a" volumeName="kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j" seLinuxMountContext="" Oct 31 02:55:22.220202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220205 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7" volumeName="kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220218 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="371390b5-879d-403c-b7e5-58bbceb94ba0" volumeName="kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220232 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a65a6ce-b6f6-4b43-a4f1-4848a347ce36" volumeName="kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220248 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="833330df-2c89-48b4-9c1d-c049b3378e26" volumeName="kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220260 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe728edf-d50f-4c7b-b98c-a12c19c39fa6" volumeName="kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220294 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8246785f-965d-4107-9e6e-1bed982b3d0e" volumeName="kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220332 4556 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec" volumeName="kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret" seLinuxMountContext="" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220344 4556 reconstruct.go:97] "Volume reconstruction finished" Oct 31 02:55:22.221027 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.220351 4556 reconciler.go:29] "Reconciler: start to sync state" Oct 31 02:55:22.225488 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.225449 4556 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 02:55:22.225608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.225567 4556 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:22.225789 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.225770 4556 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:22.227103 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.227078 4556 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:22.234627 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.234592 4556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 02:55:22.249430 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.249396 4556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 02:55:22.249430 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.249427 4556 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 02:55:22.249554 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.249445 4556 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 02:55:22.249554 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.249476 4556 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 02:55:22.252017 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.251995 4556 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259835 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="4c93a08b11bc1058dcc80be497bef8ad479d8e9c56ebfafd0340538bd2c5b51e" exitCode=0 Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259859 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="26c6b8aaf0fe41d243c3471e214fc209e89baf60da13a40ae09634c05f087c24" exitCode=0 Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259866 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="2e600a0cba4b859501a0b003f08425962f32ce4825e6807e7e06f6eaf39c9e7f" exitCode=0 Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259872 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="c21f21bc3e5fddf298f7c6b818693ec66fb079269430046ade6ffd6d0cba417c" exitCode=0 Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259880 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="ff13fe50e5b6efce8415bb60dc9df65eb156d3ef71bf4fe237f2b018e04eb024" exitCode=0 Oct 31 02:55:22.259899 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.259885 4556 generic.go:356] "Generic (PLEG): container finished" podID="cb3abcfe-0522-4dfb-abda-7f0707b65a65" containerID="064836b463f3722ca882deb15696f2fe878fb938fb0fff3b2e75d0d4e5b8db9f" exitCode=0 Oct 31 02:55:22.270365 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.270339 4556 generic.go:356] "Generic (PLEG): container finished" podID="ce9366a4079c21e3a2e7683bc4a50b18" containerID="02665be90f8ef455efa93828ffde36ea5c2426afad6ff2350a0bf7eeea3f6242" exitCode=0 Oct 31 02:55:22.294059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.294038 4556 manager.go:324] Recovery completed Oct 31 02:55:22.309377 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.309355 4556 cpu_manager.go:226] "Starting" policy="none" Oct 31 02:55:22.309377 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.309368 4556 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 02:55:22.309515 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.309385 4556 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 02:55:22.309545 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.309526 4556 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 02:55:22.310012 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.309537 4556 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 02:55:22.310065 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310015 4556 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 02:55:22.310065 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310021 4556 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 02:55:22.310065 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310028 4556 policy_none.go:50] "Start" Oct 31 02:55:22.310065 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310048 4556 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 02:55:22.310065 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310061 4556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 02:55:22.310214 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310202 4556 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 02:55:22.310243 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310215 4556 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 02:55:22.310243 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.310221 4556 policy_none.go:44] "Start" Oct 31 02:55:22.314470 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.314437 4556 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 02:55:22.318283 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318264 4556 manager.go:341] "Starting Device Plugin manager" Oct 31 02:55:22.318371 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.318295 4556 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 02:55:22.318371 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318308 4556 server.go:87] "Starting device plugin registration server" Oct 31 02:55:22.318671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318637 4556 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 02:55:22.318671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318648 4556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 02:55:22.318843 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318830 4556 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 02:55:22.320177 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318940 4556 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 02:55:22.320177 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318979 4556 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 02:55:22.320177 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.318985 4556 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 02:55:22.320177 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.319030 4556 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T02:55:22.318968688Z","UUID":"8a6c009c-8f24-400f-a555-0742314c609a","Handler":null,"Name":"","Endpoint":""} Oct 31 02:55:22.320478 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.320458 4556 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 02:55:22.321562 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.320878 4556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 02:55:22.321562 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.320903 4556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 02:55:22.350493 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.350445 4556 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal"] Oct 31 02:55:22.350948 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.350917 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"01b1495007bcae44e02a368d0e145561","Type":"ContainerStarted","Data":"eaef77661445882027f2268d6d34a9720dd0db7949282e29287b10dac683db2e"} Oct 31 02:55:22.351061 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.350958 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"01b1495007bcae44e02a368d0e145561","Type":"ContainerStarted","Data":"1810b1f0eb93db00a585a8ad232a7e40ce0f86eb82d0b253a28de6d20ddd7188"} Oct 31 02:55:22.351061 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.351008 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerStarted","Data":"8d44d426ce79cb76ccf2042113c81a701b73869c12a000efbbc5bd12378f62da"} Oct 31 02:55:22.351061 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.351010 4556 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.351061 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.351027 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerDied","Data":"02665be90f8ef455efa93828ffde36ea5c2426afad6ff2350a0bf7eeea3f6242"} Oct 31 02:55:22.351061 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.351045 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" event={"ID":"ce9366a4079c21e3a2e7683bc4a50b18","Type":"ContainerStarted","Data":"2f56b4725f4cb8397ebe9b2b1a2a647e64b3e177c68b0bf537b461dad6d4fe18"} Oct 31 02:55:22.351296 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.351143 4556 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.360681 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.360659 4556 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:22.360781 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.360705 4556 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.361277 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.361259 4556 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:22.361327 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:22.361299 4556 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.419081 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.419047 4556 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 02:55:22.420502 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.420480 4556 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 02:55:22.420617 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.420517 4556 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 02:55:22.420617 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.420533 4556 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 02:55:22.420617 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.420577 4556 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.425653 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.425632 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.425708 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.425670 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.425708 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.425700 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.430497 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.430478 4556 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.430588 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.430548 4556 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526172 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526094 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526172 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526132 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526329 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526188 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526329 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526193 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526329 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526265 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9366a4079c21e3a2e7683bc4a50b18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"ce9366a4079c21e3a2e7683bc4a50b18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:22.526329 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:22.526317 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01b1495007bcae44e02a368d0e145561-config\") pod \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" (UID: \"01b1495007bcae44e02a368d0e145561\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:23.205390 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.205355 4556 apiserver.go:52] "Watching apiserver" Oct 31 02:55:23.211248 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.211228 4556 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 02:55:23.211657 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.211639 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw","openshift-dns/dns-default-x6x7z","openshift-dns/node-resolver-k45bw","openshift-ingress-canary/ingress-canary-x7ngz","openshift-multus/network-metrics-daemon-29cdk","openshift-network-operator/iptables-alerter-2zf5j","openshift-multus/multus-additional-cni-plugins-mhr97","kube-system/global-pull-secret-syncer-j8dxn","openshift-ovn-kubernetes/ovnkube-node-7ngch","openshift-image-registry/node-ca-wjn6d","openshift-insights/insights-runtime-extractor-kkthd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-9pj75","openshift-insights/periodic-gathering-h2c2s-fwrxw","openshift-service-ca/service-ca-5fffc97bdf-jp78g","kube-system/konnectivity-agent-chd4v","openshift-network-diagnostics/network-check-target-5q2kq","openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258","openshift-multus/multus-6t7rb"] Oct 31 02:55:23.215539 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.215512 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.217589 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.217528 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 02:55:23.217939 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.217601 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wr2d\"" Oct 31 02:55:23.218363 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218333 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 02:55:23.218363 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218335 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.218671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218530 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.218671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218606 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.218671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218637 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 02:55:23.218671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218669 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.218834 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218700 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-49z5f\"" Oct 31 02:55:23.218834 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218718 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.219003 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.218984 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 02:55:23.219116 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219036 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 02:55:23.219228 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219165 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 02:55:23.219296 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219243 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7qc7w\"" Oct 31 02:55:23.219296 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219268 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-x5rg4\"" Oct 31 02:55:23.219387 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219315 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.219553 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219528 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.219630 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219618 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.219820 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.219802 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.220271 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220253 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.220512 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220486 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.220577 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220566 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 02:55:23.220671 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220618 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 02:55:23.220718 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220681 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sts47\"" Oct 31 02:55:23.220771 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220752 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.220771 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220758 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rkjm6\"" Oct 31 02:55:23.220892 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.220808 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 02:55:23.223302 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.223287 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvhtj\"" Oct 31 02:55:23.223548 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.223533 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls2x2\"" Oct 31 02:55:23.227015 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.226999 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 02:55:23.227270 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227249 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 02:55:23.227270 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227262 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 02:55:23.227364 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227279 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wcv7f\"" Oct 31 02:55:23.227364 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227291 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2b67f\"" Oct 31 02:55:23.227364 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227327 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.227527 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.227509 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.228080 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228054 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 02:55:23.228391 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228376 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbpkr\"" Oct 31 02:55:23.228654 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228499 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 02:55:23.228654 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228524 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 02:55:23.228654 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228571 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x8dcs\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228707 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zp7wq\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228713 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w5s96\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228745 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jsfq8\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228764 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hnld2\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228818 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 02:55:23.228923 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.228903 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sgkvt\"" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229016 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229089 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229114 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229155 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229174 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229198 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229177 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.229272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229252 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229276 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229308 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229344 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6ln\" (UniqueName: \"kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229362 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229390 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229406 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229456 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvkt\" (UniqueName: \"kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229489 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229512 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229528 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229551 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229586 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229390 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovn-node-metrics-cert\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229601 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.229774 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229618 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm24j\" (UniqueName: \"kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229875 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229885 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229915 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.229979 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230030 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230072 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230115 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230140 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230168 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230193 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230215 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230240 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230305 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230308 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-iptables-alerter-script\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230315 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-multus-daemon-config\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230330 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230351 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230319 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230361 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230401 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230442 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230458 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230463 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230505 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230523 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230539 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230554 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-config\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230565 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230320 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833330df-2c89-48b4-9c1d-c049b3378e26-cni-binary-copy\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230467 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230583 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230600 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230607 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230609 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-tmp\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230645 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230676 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230731 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230766 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230811 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230837 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.230954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.230853 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-tuned\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231142 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csml\" (UniqueName: \"kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231208 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231342 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd57deac-c763-4eb7-9010-8d38ea97766c-serviceca\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231402 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231474 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231736 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231769 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231803 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231834 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kb8c\" (UniqueName: \"kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231865 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231898 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231925 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231963 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.231996 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232029 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljwr\" (UniqueName: \"kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232060 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232086 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232132 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232165 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232635 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-env-overrides\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.233168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.232892 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-ovnkube-script-lib\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.236097 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.235958 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 02:55:23.236097 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.235974 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 02:55:23.236097 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.236026 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 02:55:23.236523 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.236104 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 02:55:23.236523 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.236160 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q5ns6\"" Oct 31 02:55:23.243000 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.242975 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm24j\" (UniqueName: \"kubernetes.io/projected/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-kube-api-access-pm24j\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.243088 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.243035 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6ln\" (UniqueName: \"kubernetes.io/projected/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-kube-api-access-kt6ln\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.243088 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.243035 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvkt\" (UniqueName: \"kubernetes.io/projected/833330df-2c89-48b4-9c1d-c049b3378e26-kube-api-access-dxvkt\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.243678 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.243659 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csml\" (UniqueName: \"kubernetes.io/projected/22743d27-b50f-4219-a8b7-198e417780ee-kube-api-access-4csml\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.244867 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.244849 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq42q\" (UniqueName: \"kubernetes.io/projected/636e54a6-ea98-41b1-9f02-d11a89ad344f-kube-api-access-gq42q\") pod \"network-check-target-5q2kq\" (UID: \"636e54a6-ea98-41b1-9f02-d11a89ad344f\") " pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:23.245907 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.245887 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljwr\" (UniqueName: \"kubernetes.io/projected/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-kube-api-access-5ljwr\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.245907 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.245903 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kb8c\" (UniqueName: \"kubernetes.io/projected/dd57deac-c763-4eb7-9010-8d38ea97766c-kube-api-access-2kb8c\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.286830 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.286799 4556 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:23.286998 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.286884 4556 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:23.315845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.315816 4556 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:23.315976 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.315865 4556 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:23.316786 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.316766 4556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 02:55:23.318583 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.318564 4556 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 02:55:23.318663 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.318602 4556 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-52.us-east-2.compute.internal" Oct 31 02:55:23.332637 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332615 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.332709 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332646 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.332709 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332665 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.332709 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332680 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pswzg\" (UniqueName: \"kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.332709 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332705 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlb8h\" (UniqueName: \"kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h\") pod \"migrator-66df5d496b-5p258\" (UID: \"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:23.332915 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332776 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-socket-dir-parent\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333004 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.332986 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.333059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333017 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tzx\" (UniqueName: \"kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.333059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333033 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333052 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.333191 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333100 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-var-lib-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333191 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333109 4556 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 02:55:23.333191 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333104 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aef181b7-b312-4f41-b898-be29316d5638-data-volume\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.333191 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333151 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333191 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333175 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333200 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-modprobe-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333215 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333224 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333245 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-etc-kubernetes\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333244 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkd9q\" (UniqueName: \"kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333278 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333305 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333329 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333355 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333378 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.333441 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333403 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333449 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333473 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333487 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-multus\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333502 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333492 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-cnibin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333545 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-systemd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333592 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/371390b5-879d-403c-b7e5-58bbceb94ba0-tmp-dir\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333584 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333621 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333637 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333697 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333721 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333755 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333776 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333792 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333811 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333833 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-original-pull-secret\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333846 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333852 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-system-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333856 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-k8s-cni-cncf-io\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333872 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333893 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-kubelet\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333896 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.333920 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333931 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333954 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.333985 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-run\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334005 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334014 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334025 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334059 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334084 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334123 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334148 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334151 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-sys\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334172 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334197 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334209 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334221 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334237 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8246785f-965d-4107-9e6e-1bed982b3d0e-konnectivity-ca\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334246 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334270 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334221 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aef181b7-b312-4f41-b898-be29316d5638-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334318 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd57deac-c763-4eb7-9010-8d38ea97766c-host\") pod \"node-ca-wjn6d\" (UID: \"dd57deac-c763-4eb7-9010-8d38ea97766c\") " pod="openshift-image-registry/node-ca-wjn6d" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334329 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-cni-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334368 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334435 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334449 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-key\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334356 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8246785f-965d-4107-9e6e-1bed982b3d0e-agent-certs\") pod \"konnectivity-agent-chd4v\" (UID: \"8246785f-965d-4107-9e6e-1bed982b3d0e\") " pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334476 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbv4\" (UniqueName: \"kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334500 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5248cdbd-3ee6-43c8-826b-0314c2c7644e-signing-cabundle\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334503 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334572 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqzc\" (UniqueName: \"kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334602 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334626 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334676 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-netns\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334692 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/191190f6-9b6d-498d-b960-67437ac0b983-cert\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334728 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-var-lib-kubelet\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334786 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334807 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334823 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334845 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334865 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334880 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.334868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334883 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-hostroot\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334896 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lzn\" (UniqueName: \"kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334929 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-etc-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334949 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-host\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334971 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-bin\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.334978 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335002 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335009 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c89547-d1bb-4dfa-94e5-cd303ed05f2a-host-slash\") pod \"iptables-alerter-2zf5j\" (UID: \"12c89547-d1bb-4dfa-94e5-cd303ed05f2a\") " pod="openshift-network-operator/iptables-alerter-2zf5j" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335025 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335049 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335057 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-kubernetes\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335074 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335091 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-var-lib-cni-bin\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335098 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335121 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-host-run-multus-certs\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335134 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-ovn\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335124 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335170 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335158 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-run-openvswitch\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335198 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335223 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335226 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-systemd\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335244 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335268 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335293 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335310 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335316 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335337 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335379 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-systemd-units\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335381 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-tmp-dir\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335379 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335430 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-kubelet\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335436 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-slash\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335438 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335488 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335492 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-log-socket\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335497 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-node-log\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335456 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-cni-netd\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335516 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8xg\" (UniqueName: \"kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335541 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335562 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-d\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335639 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335567 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833330df-2c89-48b4-9c1d-c049b3378e26-multus-conf-dir\") pod \"multus-6t7rb\" (UID: \"833330df-2c89-48b4-9c1d-c049b3378e26\") " pod="openshift-multus/multus-6t7rb" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335679 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335689 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22743d27-b50f-4219-a8b7-198e417780ee-metrics-certs\") pod \"network-metrics-daemon-29cdk\" (UID: \"22743d27-b50f-4219-a8b7-198e417780ee\") " pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335681 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysctl-conf\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335708 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335748 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335768 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335784 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335844 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7-host-run-netns\") pod \"ovnkube-node-7ngch\" (UID: \"1cc5c3ae-2635-4f11-a2fa-cc14fbb02db7\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335876 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-etc-sysconfig\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335909 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-config-volume\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.336272 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.335952 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe728edf-d50f-4c7b-b98c-a12c19c39fa6-lib-modules\") pod \"tuned-9pj75\" (UID: \"fe728edf-d50f-4c7b-b98c-a12c19c39fa6\") " pod="openshift-cluster-node-tuning-operator/tuned-9pj75" Oct 31 02:55:23.362117 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362086 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlb8h\" (UniqueName: \"kubernetes.io/projected/4a65a6ce-b6f6-4b43-a4f1-4848a347ce36-kube-api-access-dlb8h\") pod \"migrator-66df5d496b-5p258\" (UID: \"4a65a6ce-b6f6-4b43-a4f1-4848a347ce36\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5p258" Oct 31 02:55:23.362117 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362102 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbv4\" (UniqueName: \"kubernetes.io/projected/cb3abcfe-0522-4dfb-abda-7f0707b65a65-kube-api-access-5wbv4\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.362396 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362331 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkd9q\" (UniqueName: \"kubernetes.io/projected/5248cdbd-3ee6-43c8-826b-0314c2c7644e-kube-api-access-tkd9q\") pod \"service-ca-5fffc97bdf-jp78g\" (UID: \"5248cdbd-3ee6-43c8-826b-0314c2c7644e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jp78g" Oct 31 02:55:23.362396 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362343 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"periodic-gathering-h2c2s-fwrxw\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:55:23.362396 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362333 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8xg\" (UniqueName: \"kubernetes.io/projected/371390b5-879d-403c-b7e5-58bbceb94ba0-kube-api-access-4n8xg\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.362604 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362505 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjn6d_dd57deac-c763-4eb7-9010-8d38ea97766c/node-ca/0.log" Oct 31 02:55:23.362750 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362730 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pswzg\" (UniqueName: \"kubernetes.io/projected/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-kube-api-access-pswzg\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.362812 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362733 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lzn\" (UniqueName: \"kubernetes.io/projected/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kube-api-access-w2lzn\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.362881 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.362859 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqzc\" (UniqueName: \"kubernetes.io/projected/191190f6-9b6d-498d-b960-67437ac0b983-kube-api-access-bjqzc\") pod \"ingress-canary-x7ngz\" (UID: \"191190f6-9b6d-498d-b960-67437ac0b983\") " pod="openshift-ingress-canary/ingress-canary-x7ngz" Oct 31 02:55:23.364989 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.364967 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tzx\" (UniqueName: \"kubernetes.io/projected/aef181b7-b312-4f41-b898-be29316d5638-kube-api-access-j2tzx\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.436317 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436284 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436317 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436315 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436330 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436349 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436376 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436406 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436438 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-device-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436459 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-kubelet-config\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436441 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aef181b7-b312-4f41-b898-be29316d5638-crio-socket\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436469 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436493 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436527 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436542 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436545 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436567 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436591 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436602 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-registration-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436612 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec-dbus\") pod \"global-pull-secret-syncer-j8dxn\" (UID: \"b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec\") " pod="kube-system/global-pull-secret-syncer-j8dxn" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436605 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436645 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436652 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/371390b5-879d-403c-b7e5-58bbceb94ba0-hosts-file\") pod \"node-resolver-k45bw\" (UID: \"371390b5-879d-403c-b7e5-58bbceb94ba0\") " pod="openshift-dns/node-resolver-k45bw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436662 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436698 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436706 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-system-cni-dir\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.436707 4556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436717 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436735 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436747 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-socket-dir\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436703 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb3abcfe-0522-4dfb-abda-7f0707b65a65-cnibin\") pod \"multus-additional-cni-plugins-mhr97\" (UID: \"cb3abcfe-0522-4dfb-abda-7f0707b65a65\") " pod="openshift-multus/multus-additional-cni-plugins-mhr97" Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.436789 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.936757721 +0000 UTC m=+1.791774840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.436854 4556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.436903 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:23.936886384 +0000 UTC m=+1.791903516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.436940 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.436924 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1d8733a-ce73-455e-aeb6-bba45d34b97f-sys-fs\") pod \"aws-ebs-csi-driver-node-6zbtw\" (UID: \"c1d8733a-ce73-455e-aeb6-bba45d34b97f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" Oct 31 02:55:23.515853 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.515774 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29cdk" Oct 31 02:55:23.623262 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.623229 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:23.772868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.772785 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:23.808323 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.808278 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29cdk"] Oct 31 02:55:23.812161 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:23.812134 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22743d27_b50f_4219_a8b7_198e417780ee.slice/crio-a136a161396ea99e914da54ef6f3896a3f530d3ce55aa4d94251c2c90ea35e1a WatchSource:0}: Error finding container a136a161396ea99e914da54ef6f3896a3f530d3ce55aa4d94251c2c90ea35e1a: Status 404 returned error can't find the container with id a136a161396ea99e914da54ef6f3896a3f530d3ce55aa4d94251c2c90ea35e1a Oct 31 02:55:23.814822 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.814805 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:55:23.927607 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.927569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:23.940250 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.940215 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:23.940383 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:23.940276 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:23.940458 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.940382 4556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:23.940458 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.940398 4556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.940560 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.940480 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:24.940462553 +0000 UTC m=+2.795479690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:23.940560 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:23.940516 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:24.940496539 +0000 UTC m=+2.795513767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:24.064482 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.064377 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:24.217474 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.217433 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:24.289836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.289803 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29cdk" event={"ID":"22743d27-b50f-4219-a8b7-198e417780ee","Type":"ContainerStarted","Data":"a136a161396ea99e914da54ef6f3896a3f530d3ce55aa4d94251c2c90ea35e1a"} Oct 31 02:55:24.342022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.341944 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7ngz_191190f6-9b6d-498d-b960-67437ac0b983/serve-healthcheck-canary/0.log" Oct 31 02:55:24.546927 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.546898 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5p258_4a65a6ce-b6f6-4b43-a4f1-4848a347ce36/migrator/0.log" Oct 31 02:55:24.711530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.711492 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:24.791176 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.791136 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:24.850184 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.850156 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:24.948356 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.948331 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:24.948512 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:24.948388 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:24.948512 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:24.948501 4556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:24.948587 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:24.948512 4556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:24.948587 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:24.948554 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:26.948541565 +0000 UTC m=+4.803558685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:24.948587 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:24.948578 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:26.94856141 +0000 UTC m=+4.803578532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:25.295473 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:25.295365 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29cdk" event={"ID":"22743d27-b50f-4219-a8b7-198e417780ee","Type":"ContainerStarted","Data":"04413a319e6f37240cdef958fdcce7542feae16349a7e25a2652cd4354888f85"} Oct 31 02:55:25.295473 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:25.295438 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29cdk" event={"ID":"22743d27-b50f-4219-a8b7-198e417780ee","Type":"ContainerStarted","Data":"286f7f9585e4f542f46887bde474a4bc2d154866a5dd53b584794d13f17e99f6"} Oct 31 02:55:25.417006 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:25.416969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:25.618505 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:25.618424 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:25.806759 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:25.806730 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:26.002521 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:26.002488 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:26.208702 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:26.208664 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:26.512155 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:26.512118 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:26.963550 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:26.963512 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:26.963720 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:26.963564 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:26.963720 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:26.963670 4556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 02:55:26.963805 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:26.963743 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls podName:ea5034e0-53b0-48d6-8d9e-71c8eb577ca4 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:30.963729158 +0000 UTC m=+8.818746276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls") pod "dns-default-x6x7z" (UID: "ea5034e0-53b0-48d6-8d9e-71c8eb577ca4") : secret "dns-default-metrics-tls" not found Oct 31 02:55:26.963805 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:26.963672 4556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 02:55:26.963805 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:26.963780 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls podName:aef181b7-b312-4f41-b898-be29316d5638 nodeName:}" failed. No retries permitted until 2026-10-31 02:55:30.963773572 +0000 UTC m=+8.818790732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kkthd" (UID: "aef181b7-b312-4f41-b898-be29316d5638") : secret "insights-runtime-extractor-tls" not found Oct 31 02:55:27.421239 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:27.421208 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:27.561696 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:27.561655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:27.687482 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:27.687394 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:27.819390 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:27.819359 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:28.005607 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:28.005531 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:28.208504 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:28.208454 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:29.410991 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.410951 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:29.610376 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.610342 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:29.748768 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.748699 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:29.748943 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.748930 4556 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 02:55:29.779224 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.779195 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:29.914406 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:29.914372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:30.049070 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.048990 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:30.205337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.205303 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:30.814466 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.814429 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:30.896727 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.896695 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ngch" Oct 31 02:55:30.992983 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.992941 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:30.993166 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.993073 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:30.995444 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.995401 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aef181b7-b312-4f41-b898-be29316d5638-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkthd\" (UID: \"aef181b7-b312-4f41-b898-be29316d5638\") " pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:30.995580 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:30.995402 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea5034e0-53b0-48d6-8d9e-71c8eb577ca4-metrics-tls\") pod \"dns-default-x6x7z\" (UID: \"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4\") " pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:31.008993 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.008959 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:31.021017 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.020989 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:31.021156 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.021031 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkthd" Oct 31 02:55:31.300457 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.300357 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6x7z"] Oct 31 02:55:31.303625 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.303596 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkthd"] Oct 31 02:55:31.304802 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:31.304775 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5034e0_53b0_48d6_8d9e_71c8eb577ca4.slice/crio-35ec52b4c63cdd2a0af6ec00f97bcb3ea7a40ae5e66dabfb3b4d8c62897a0925 WatchSource:0}: Error finding container 35ec52b4c63cdd2a0af6ec00f97bcb3ea7a40ae5e66dabfb3b4d8c62897a0925: Status 404 returned error can't find the container with id 35ec52b4c63cdd2a0af6ec00f97bcb3ea7a40ae5e66dabfb3b4d8c62897a0925 Oct 31 02:55:31.318805 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.318771 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkthd" event={"ID":"aef181b7-b312-4f41-b898-be29316d5638","Type":"ContainerStarted","Data":"b5fed2ac0dbd53b83450fa798a51d72c67bc19e04a8d175c257615b28ec448c5"} Oct 31 02:55:31.319954 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:31.319931 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6x7z" event={"ID":"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4","Type":"ContainerStarted","Data":"35ec52b4c63cdd2a0af6ec00f97bcb3ea7a40ae5e66dabfb3b4d8c62897a0925"} Oct 31 02:55:32.326007 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:32.325906 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkthd" event={"ID":"aef181b7-b312-4f41-b898-be29316d5638","Type":"ContainerStarted","Data":"d61445060fbe51d1b739a8ccccedbe0a9f7cf90a7bd051060a578cee493df36a"} Oct 31 02:55:32.326007 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:32.325970 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkthd" event={"ID":"aef181b7-b312-4f41-b898-be29316d5638","Type":"ContainerStarted","Data":"46bdaf04e7e87e67dae316145bd46b54a2ccb62fdb8f4a6d637d32283999bb07"} Oct 31 02:55:33.332878 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:33.332789 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6x7z" event={"ID":"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4","Type":"ContainerStarted","Data":"6e7aa02dc36043c7695ed26be1f8b7093a9edc3c1df2037910bc63b9ecf190ae"} Oct 31 02:55:33.333305 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:33.332896 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6x7z" event={"ID":"ea5034e0-53b0-48d6-8d9e-71c8eb577ca4","Type":"ContainerStarted","Data":"88c0e8a08b564baa2a8aea7769a04a1fc6045e397ed9475669760f6a00d4d025"} Oct 31 02:55:33.705585 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:33.705553 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chd4v" Oct 31 02:55:34.340183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:34.340141 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkthd" event={"ID":"aef181b7-b312-4f41-b898-be29316d5638","Type":"ContainerStarted","Data":"e616bce181b75f52c0625f95204571776d08526a48a6e1db61fcbfd3dfe9194e"} Oct 31 02:55:34.340563 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:34.340438 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:39.160745 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.160709 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d5d5b459f-d5zvx"] Oct 31 02:55:39.163967 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.163944 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.168018 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.167968 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 02:55:39.169775 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.169751 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 02:55:39.169877 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.169749 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 02:55:39.169877 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.169780 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sc997\"" Oct 31 02:55:39.169877 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.169836 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 02:55:39.182750 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.182726 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5d5b459f-d5zvx"] Oct 31 02:55:39.200378 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.200342 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:39.246690 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246658 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-certificates\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.246690 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246692 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrr8b\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-kube-api-access-vrr8b\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.246942 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246718 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-image-registry-private-configuration\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.246942 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246839 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-bound-sa-token\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.246942 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246897 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca616b97-e45b-49e3-8ee3-85af2a334fc3-ca-trust-extracted\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.246942 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246931 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-trusted-ca\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.247130 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.246968 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-installation-pull-secrets\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.247130 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.247026 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-tls\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348246 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348213 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca616b97-e45b-49e3-8ee3-85af2a334fc3-ca-trust-extracted\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348246 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348246 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-trusted-ca\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348269 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-installation-pull-secrets\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348300 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-tls\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348331 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-certificates\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348363 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrr8b\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-kube-api-access-vrr8b\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348400 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-image-registry-private-configuration\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348452 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-bound-sa-token\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.348900 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.348878 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ca616b97-e45b-49e3-8ee3-85af2a334fc3-ca-trust-extracted\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.349468 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.349445 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-trusted-ca\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.349468 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.349458 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-certificates\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.351033 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.351010 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-installation-pull-secrets\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.351127 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.351050 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ca616b97-e45b-49e3-8ee3-85af2a334fc3-image-registry-private-configuration\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.351166 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.351138 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-registry-tls\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.360583 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.360552 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrr8b\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-kube-api-access-vrr8b\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.360660 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.360562 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca616b97-e45b-49e3-8ee3-85af2a334fc3-bound-sa-token\") pod \"image-registry-5d5d5b459f-d5zvx\" (UID: \"ca616b97-e45b-49e3-8ee3-85af2a334fc3\") " pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.370322 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.370298 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:39.475985 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.475888 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:39.723377 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.723336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:39.753804 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.753727 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d5d5b459f-d5zvx"] Oct 31 02:55:39.758162 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:39.758127 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca616b97_e45b_49e3_8ee3_85af2a334fc3.slice/crio-b9621c720f6ad358376787118bc8e93cfb93c87b3672307856974a129c5caacc WatchSource:0}: Error finding container b9621c720f6ad358376787118bc8e93cfb93c87b3672307856974a129c5caacc: Status 404 returned error can't find the container with id b9621c720f6ad358376787118bc8e93cfb93c87b3672307856974a129c5caacc Oct 31 02:55:39.910323 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:39.910278 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:40.359649 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:40.359614 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" event={"ID":"ca616b97-e45b-49e3-8ee3-85af2a334fc3","Type":"ContainerStarted","Data":"770b948a39a606c10159950a985d0d7e462729ab340e273539e2557c48ce7cec"} Oct 31 02:55:40.360062 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:40.359659 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" event={"ID":"ca616b97-e45b-49e3-8ee3-85af2a334fc3","Type":"ContainerStarted","Data":"b9621c720f6ad358376787118bc8e93cfb93c87b3672307856974a129c5caacc"} Oct 31 02:55:40.360062 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:40.359756 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:55:40.383031 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:40.382966 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" podStartSLOduration=1.382952857 podStartE2EDuration="1.382952857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:40.382121899 +0000 UTC m=+18.237139040" watchObservedRunningTime="2026-10-31 02:55:40.382952857 +0000 UTC m=+18.237970007" Oct 31 02:55:41.937493 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:41.937462 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:42.069641 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:42.069593 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:44.070959 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:44.070923 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:44.345574 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:44.345502 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6x7z" Oct 31 02:55:45.933875 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:55:45.933828 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:55:46.093976 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.093945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:46.308139 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.308041 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9lk6f"] Oct 31 02:55:46.323979 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.323936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:46.342147 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.342122 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.386069 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.386037 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 02:55:46.386236 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.386216 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtdnh\"" Oct 31 02:55:46.386236 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.386221 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 02:55:46.395524 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395496 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-sys\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395524 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395528 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p962\" (UniqueName: \"kubernetes.io/projected/f32d48d6-7d90-4329-86a4-6243beb7f9fd-kube-api-access-7p962\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395680 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395556 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395680 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395638 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-metrics-client-ca\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395755 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395707 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-wtmp\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395755 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395743 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395830 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395815 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-textfile\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395869 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-root\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.395900 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.395870 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-tls\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.418599 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.418563 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 02:55:46.418724 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.418604 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 02:55:46.418767 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.418741 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 02:55:46.418837 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.418824 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 02:55:46.476814 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.476784 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:46.496814 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496788 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-root\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.496915 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496825 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-tls\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.496915 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496848 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-sys\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.496915 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496867 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p962\" (UniqueName: \"kubernetes.io/projected/f32d48d6-7d90-4329-86a4-6243beb7f9fd-kube-api-access-7p962\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496910 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-root\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496911 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-sys\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.496986 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497032 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-metrics-client-ca\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497244 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497062 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-wtmp\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497244 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497148 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497244 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497188 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-wtmp\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497244 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497189 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-textfile\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497468 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497450 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-textfile\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497685 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497654 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.497764 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.497664 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f32d48d6-7d90-4329-86a4-6243beb7f9fd-metrics-client-ca\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.499277 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.499252 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-tls\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.499444 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.499406 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f32d48d6-7d90-4329-86a4-6243beb7f9fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.510470 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.510382 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p962\" (UniqueName: \"kubernetes.io/projected/f32d48d6-7d90-4329-86a4-6243beb7f9fd-kube-api-access-7p962\") pod \"node-exporter-9lk6f\" (UID: \"f32d48d6-7d90-4329-86a4-6243beb7f9fd\") " pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.690630 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.690587 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lk6f" Oct 31 02:55:46.709214 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.709179 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:46.722958 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:46.722931 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32d48d6_7d90_4329_86a4_6243beb7f9fd.slice/crio-30ba7f6cd299abcdc2301d2f2941c6c2e7e5d039590c71da15fa45b1e2cfdaa9 WatchSource:0}: Error finding container 30ba7f6cd299abcdc2301d2f2941c6c2e7e5d039590c71da15fa45b1e2cfdaa9: Status 404 returned error can't find the container with id 30ba7f6cd299abcdc2301d2f2941c6c2e7e5d039590c71da15fa45b1e2cfdaa9 Oct 31 02:55:46.841176 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:46.841132 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:47.324579 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:47.324546 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:47.381617 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:47.381588 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lk6f" event={"ID":"f32d48d6-7d90-4329-86a4-6243beb7f9fd","Type":"ContainerStarted","Data":"30ba7f6cd299abcdc2301d2f2941c6c2e7e5d039590c71da15fa45b1e2cfdaa9"} Oct 31 02:55:47.515337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:47.515301 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:48.154608 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:48.154576 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:48.332131 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:48.332100 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:48.386424 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:48.386375 4556 generic.go:356] "Generic (PLEG): container finished" podID="f32d48d6-7d90-4329-86a4-6243beb7f9fd" containerID="1caf50fed6db17dc6d0cb651dcb0370547d288f66031d595427920eca822c682" exitCode=0 Oct 31 02:55:48.386598 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:48.386443 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lk6f" event={"ID":"f32d48d6-7d90-4329-86a4-6243beb7f9fd","Type":"ContainerDied","Data":"1caf50fed6db17dc6d0cb651dcb0370547d288f66031d595427920eca822c682"} Oct 31 02:55:48.473868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:48.473824 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:49.311429 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.311387 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:49.391426 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.391376 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lk6f" event={"ID":"f32d48d6-7d90-4329-86a4-6243beb7f9fd","Type":"ContainerStarted","Data":"54e6f52ef70902c5ec42588b705a712442e7b170762201c746c449880eccbc34"} Oct 31 02:55:49.391867 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.391476 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lk6f" event={"ID":"f32d48d6-7d90-4329-86a4-6243beb7f9fd","Type":"ContainerStarted","Data":"b059e31409ac6d2c3b2aa1c8908e7931dd6d6e652fe4f2ee6dd79eca15a9a940"} Oct 31 02:55:49.412896 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.412851 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9lk6f" podStartSLOduration=2.212909901 podStartE2EDuration="3.412838401s" totalImagesPullingTime="1.134440857s" totalInitContainerRuntime="65.487643ms" isStatefulPod=true podCreationTimestamp="2026-10-31 02:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:49.412139291 +0000 UTC m=+27.267156454" watchObservedRunningTime="2026-10-31 02:55:49.412838401 +0000 UTC m=+27.267855541" Oct 31 02:55:49.450852 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.450824 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:49.589288 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:49.589215 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:50.723143 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.723106 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-589dc7c4b9-vc6j9"] Oct 31 02:55:50.726352 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.726334 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.732846 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.732458 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 02:55:50.732846 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.732488 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 02:55:50.733275 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.733258 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 02:55:50.733462 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.733448 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 02:55:50.733505 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.733464 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s6fpw\"" Oct 31 02:55:50.733580 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.733563 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-18bjpvh54eggr\"" Oct 31 02:55:50.735450 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.735431 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:50.737730 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.737710 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-589dc7c4b9-vc6j9"] Oct 31 02:55:50.831230 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831192 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4975fc72-46ce-4ecc-87eb-500108bb4304-audit-log\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831230 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831238 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831453 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831326 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-tls\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831453 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831359 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmj4\" (UniqueName: \"kubernetes.io/projected/4975fc72-46ce-4ecc-87eb-500108bb4304-kube-api-access-8qmj4\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831453 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831390 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-metrics-server-audit-profiles\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831553 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831478 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-client-certs\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.831553 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.831499 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-client-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.874056 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.874029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:50.932695 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932664 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4975fc72-46ce-4ecc-87eb-500108bb4304-audit-log\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932782 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932706 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932840 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932821 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-tls\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932874 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932855 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmj4\" (UniqueName: \"kubernetes.io/projected/4975fc72-46ce-4ecc-87eb-500108bb4304-kube-api-access-8qmj4\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932917 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932882 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-metrics-server-audit-profiles\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932917 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932911 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-client-certs\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.932984 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.932942 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-client-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.933146 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.933122 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4975fc72-46ce-4ecc-87eb-500108bb4304-audit-log\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.933533 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.933511 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.933883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.933859 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4975fc72-46ce-4ecc-87eb-500108bb4304-metrics-server-audit-profiles\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.935244 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.935225 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-tls\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.935323 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.935263 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-client-ca-bundle\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.935495 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.935476 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4975fc72-46ce-4ecc-87eb-500108bb4304-secret-metrics-server-client-certs\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:50.944576 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:50.944557 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmj4\" (UniqueName: \"kubernetes.io/projected/4975fc72-46ce-4ecc-87eb-500108bb4304-kube-api-access-8qmj4\") pod \"metrics-server-589dc7c4b9-vc6j9\" (UID: \"4975fc72-46ce-4ecc-87eb-500108bb4304\") " pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:51.037833 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:51.037745 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:55:51.122345 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:51.122313 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:51.264007 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:51.263969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:51.312081 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:51.311992 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-589dc7c4b9-vc6j9"] Oct 31 02:55:51.315502 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:51.315473 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4975fc72_46ce_4ecc_87eb_500108bb4304.slice/crio-9a2d4105fc2a1f4a1cfb88c05b3b565b26ddc2a510da52b855670d69bb017747 WatchSource:0}: Error finding container 9a2d4105fc2a1f4a1cfb88c05b3b565b26ddc2a510da52b855670d69bb017747: Status 404 returned error can't find the container with id 9a2d4105fc2a1f4a1cfb88c05b3b565b26ddc2a510da52b855670d69bb017747 Oct 31 02:55:51.399010 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:51.398975 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" event={"ID":"4975fc72-46ce-4ecc-87eb-500108bb4304","Type":"ContainerStarted","Data":"9a2d4105fc2a1f4a1cfb88c05b3b565b26ddc2a510da52b855670d69bb017747"} Oct 31 02:55:52.555120 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.555085 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:55:52.559179 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.559153 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.568582 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568551 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 02:55:52.568582 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568568 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 02:55:52.568785 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568620 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 02:55:52.568785 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568711 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pagvvcqbttfc\"" Oct 31 02:55:52.568915 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568896 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vqvgz\"" Oct 31 02:55:52.568980 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.568952 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 02:55:52.569206 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569188 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 02:55:52.569268 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569216 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 02:55:52.569268 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569246 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 02:55:52.569360 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569298 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 02:55:52.569360 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569310 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 02:55:52.569360 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569328 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 02:55:52.569526 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.569477 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 02:55:52.571285 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.571252 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 02:55:52.571562 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.571545 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 02:55:52.571838 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.571820 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 02:55:52.580785 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.580703 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:55:52.622360 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.622325 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:52.647567 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647528 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647728 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647614 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647728 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647649 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhvj\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647728 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647699 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647728 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647717 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647885 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647801 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647885 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647844 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647885 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647864 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.647885 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647880 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647900 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647943 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.647992 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648007 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648020 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648228 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648037 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648228 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648051 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648228 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648110 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648228 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648147 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648385 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648234 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.648385 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.648262 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749670 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749634 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749670 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749676 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749698 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhvj\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749724 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749751 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749791 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749820 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749845 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749868 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.749922 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749895 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749925 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749956 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.749983 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750007 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750037 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750065 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750095 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750119 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750155 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750198 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750337 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750327 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.750905 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.750349 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.752713 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.751002 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.752713 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.751079 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.752713 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.751546 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.752713 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.752360 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.753931 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.753897 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.754036 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.753989 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.754462 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.754435 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.756771 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.756733 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.756868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.756845 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.757322 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.757300 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.758901 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.758877 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.759857 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.759011 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.759857 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.759361 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.759857 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.759494 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.759857 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.759686 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.759857 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.759813 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.760999 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.760959 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.761377 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.761344 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhvj\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\") pod \"prometheus-k8s-0\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:52.886736 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:52.886710 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:55:53.134867 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.134788 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:53.174210 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.174180 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:55:53.177078 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:55:53.177048 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57708ca1_bb6c_4b23_aaff_988c38f99d03.slice/crio-196e9a49db9fe62a829c990b066efa7da16bd523d18036076cddc392fd1d4aa1 WatchSource:0}: Error finding container 196e9a49db9fe62a829c990b066efa7da16bd523d18036076cddc392fd1d4aa1: Status 404 returned error can't find the container with id 196e9a49db9fe62a829c990b066efa7da16bd523d18036076cddc392fd1d4aa1 Oct 31 02:55:53.278054 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.278019 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:53.406845 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.406809 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" event={"ID":"4975fc72-46ce-4ecc-87eb-500108bb4304","Type":"ContainerStarted","Data":"1e9fb8a62332508ba33f51c45606b20eb77c0d5e68669823ae518c44674491b6"} Oct 31 02:55:53.408142 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.408117 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"196e9a49db9fe62a829c990b066efa7da16bd523d18036076cddc392fd1d4aa1"} Oct 31 02:55:53.422750 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.422711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:53.425143 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.425098 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" podStartSLOduration=1.912248487 podStartE2EDuration="3.425071947s" totalImagesPullingTime="1.51282346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:55:53.425014342 +0000 UTC m=+31.280031484" watchObservedRunningTime="2026-10-31 02:55:53.425071947 +0000 UTC m=+31.280089088" Oct 31 02:55:53.562479 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:53.562441 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:54.818846 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:54.818821 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:54.967257 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:54.967225 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:55.415687 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:55.415655 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="df09b50ba2842d960b3779d192fd0e2a799e6d8aa64982899a24388461b87373" exitCode=0 Oct 31 02:55:55.415836 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:55.415742 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"df09b50ba2842d960b3779d192fd0e2a799e6d8aa64982899a24388461b87373"} Oct 31 02:55:55.610614 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:55.610583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:55.776436 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:55.776346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:55:56.515906 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:56.515872 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5q2kq" Oct 31 02:55:59.433235 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:59.433187 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"f209a5f5046486072b3f30038a640e97711054d8701548a3593bb484e3f726ce"} Oct 31 02:55:59.433749 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:59.433257 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"7c3e5603109c93a379725a1085607436bdd7205af407a5c5fa75dcaee8906a95"} Oct 31 02:55:59.894883 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:55:59.894792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:00.039459 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:00.039424 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:01.367293 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.367266 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d5d5b459f-d5zvx" Oct 31 02:56:01.444621 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.444583 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"cc0b25ff07ef54e1b1774a0bb4bcf7cc7382d1b13130eae9a0c587bb20074752"} Oct 31 02:56:01.444791 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.444643 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"b442c59a3b4642c21c701395a10f8156d744a364c4f5e9aa3037e91331dacfac"} Oct 31 02:56:01.444791 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.444676 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"ee5fc2aeebfe98f4b8623947e966cac451df07d00c2a6ca9e185cd0ef21986cd"} Oct 31 02:56:01.444791 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.444700 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerStarted","Data":"2cb0f579826551f4c200bb054ba6b131428472a9b6d0a6bae5b695c2e7bc12da"} Oct 31 02:56:01.485773 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.485719 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.74040599 podStartE2EDuration="9.485701853s" totalImagesPullingTime="6.715887327s" totalInitContainerRuntime="29.408536ms" isStatefulPod=false podCreationTimestamp="2026-10-31 02:55:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:01.484402619 +0000 UTC m=+39.339419759" watchObservedRunningTime="2026-10-31 02:56:01.485701853 +0000 UTC m=+39.340718994" Oct 31 02:56:01.602283 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.602242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:01.749184 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:01.749142 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:02.594805 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:02.594770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:02.733853 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:02.733818 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:02.887741 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:02.887675 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:03.346091 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:03.346055 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:03.363780 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:03.363755 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:03.469058 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:03.469033 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:05.233617 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:05.233529 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:05.372851 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:05.372813 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:06.952315 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:06.952284 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:07.121953 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:07.121911 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:09.739897 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:09.739865 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:09.904621 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:09.904584 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:11.038605 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:11.038568 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:56:11.038996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:11.038635 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:56:14.184720 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:14.184684 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:14.324435 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:14.324395 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:15.934886 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:56:15.934850 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:56:16.387162 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:16.387080 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:16.519534 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:16.519501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:18.386278 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:18.386248 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:18.526868 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:18.526838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:24.712175 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:24.712138 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:24.890458 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:24.890430 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:28.229919 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:28.229886 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:29.983984 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:29.983952 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:30.119329 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:30.119284 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:31.043703 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:31.043674 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:56:31.047609 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:31.047581 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-589dc7c4b9-vc6j9" Oct 31 02:56:35.116643 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:35.116603 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:35.246701 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:35.246669 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:35.398200 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:35.398118 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:35.525639 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:35.525610 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:38.172512 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.172474 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:56:38.173022 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.172918 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" podUID="503a089d-2343-450d-bba6-486321ca52e4" containerName="insights-gathering" containerID="cri-o://658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7" gracePeriod=30 Oct 31 02:56:38.417612 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.417584 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:56:38.511294 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511196 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") pod \"503a089d-2343-450d-bba6-486321ca52e4\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " Oct 31 02:56:38.511294 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") pod \"503a089d-2343-450d-bba6-486321ca52e4\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " Oct 31 02:56:38.511294 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511299 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") pod \"503a089d-2343-450d-bba6-486321ca52e4\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " Oct 31 02:56:38.511681 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511333 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") pod \"503a089d-2343-450d-bba6-486321ca52e4\" (UID: \"503a089d-2343-450d-bba6-486321ca52e4\") " Oct 31 02:56:38.511681 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511647 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle" pod "503a089d-2343-450d-bba6-486321ca52e4" (UID: "503a089d-2343-450d-bba6-486321ca52e4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:38.511787 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511721 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path" pod "503a089d-2343-450d-bba6-486321ca52e4" (UID: "503a089d-2343-450d-bba6-486321ca52e4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:38.511835 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.511776 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle" pod "503a089d-2343-450d-bba6-486321ca52e4" (UID: "503a089d-2343-450d-bba6-486321ca52e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:38.513599 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.513579 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx" pod "503a089d-2343-450d-bba6-486321ca52e4" (UID: "503a089d-2343-450d-bba6-486321ca52e4"). InnerVolumeSpecName "kube-api-access-d8zzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:38.564888 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.564853 4556 generic.go:356] "Generic (PLEG): container finished" podID="503a089d-2343-450d-bba6-486321ca52e4" containerID="658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7" exitCode=2 Oct 31 02:56:38.565050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.564898 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" Oct 31 02:56:38.565050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.564934 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" event={"ID":"503a089d-2343-450d-bba6-486321ca52e4","Type":"ContainerDied","Data":"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7"} Oct 31 02:56:38.565050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.564983 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-h2c2s-fwrxw" event={"ID":"503a089d-2343-450d-bba6-486321ca52e4","Type":"ContainerDied","Data":"75c04598e48138c095aa3c0c4e73978785ae20fcb4d6aa927e2f8b4025418220"} Oct 31 02:56:38.565050 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.565006 4556 scope.go:122] "RemoveContainer" containerID="658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7" Oct 31 02:56:38.573991 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.573969 4556 scope.go:122] "RemoveContainer" containerID="658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7" Oct 31 02:56:38.574291 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:56:38.574268 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7\": container with ID starting with 658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7 not found: ID does not exist" containerID="658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7" Oct 31 02:56:38.574357 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.574300 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7"} err="failed to get container status \"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7\": rpc error: code = NotFound desc = could not find container \"658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7\": container with ID starting with 658b6868e1cfc7265f21184b71e3ac1ed749e1a2afb9ab9b4c491e72812708e7 not found: ID does not exist" Oct 31 02:56:38.582971 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.582943 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:56:38.586972 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.586932 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-h2c2s-fwrxw"] Oct 31 02:56:38.612874 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.612840 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zzx\" (UniqueName: \"kubernetes.io/projected/503a089d-2343-450d-bba6-486321ca52e4-kube-api-access-d8zzx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:38.612874 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.612873 4556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-trusted-ca-bundle\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:38.613059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.612886 4556 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/503a089d-2343-450d-bba6-486321ca52e4-archives-path\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:38.613059 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:38.612896 4556 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503a089d-2343-450d-bba6-486321ca52e4-service-ca-bundle\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:40.254837 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.254799 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503a089d-2343-450d-bba6-486321ca52e4" path="/var/lib/kubelet/pods/503a089d-2343-450d-bba6-486321ca52e4/volumes" Oct 31 02:56:40.412124 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412089 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:40.412593 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412439 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cc0b25ff07ef54e1b1774a0bb4bcf7cc7382d1b13130eae9a0c587bb20074752" gracePeriod=600 Oct 31 02:56:40.412593 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412454 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-web" containerID="cri-o://ee5fc2aeebfe98f4b8623947e966cac451df07d00c2a6ca9e185cd0ef21986cd" gracePeriod=600 Oct 31 02:56:40.412593 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412491 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="config-reloader" containerID="cri-o://f209a5f5046486072b3f30038a640e97711054d8701548a3593bb484e3f726ce" gracePeriod=600 Oct 31 02:56:40.412593 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412498 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="thanos-sidecar" containerID="cri-o://2cb0f579826551f4c200bb054ba6b131428472a9b6d0a6bae5b695c2e7bc12da" gracePeriod=600 Oct 31 02:56:40.412593 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412540 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy" containerID="cri-o://b442c59a3b4642c21c701395a10f8156d744a364c4f5e9aa3037e91331dacfac" gracePeriod=600 Oct 31 02:56:40.412955 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.412398 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="prometheus" containerID="cri-o://7c3e5603109c93a379725a1085607436bdd7205af407a5c5fa75dcaee8906a95" gracePeriod=600 Oct 31 02:56:40.575220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575188 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="cc0b25ff07ef54e1b1774a0bb4bcf7cc7382d1b13130eae9a0c587bb20074752" exitCode=0 Oct 31 02:56:40.575220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575212 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="b442c59a3b4642c21c701395a10f8156d744a364c4f5e9aa3037e91331dacfac" exitCode=0 Oct 31 02:56:40.575220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575218 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="2cb0f579826551f4c200bb054ba6b131428472a9b6d0a6bae5b695c2e7bc12da" exitCode=0 Oct 31 02:56:40.575220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575224 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="f209a5f5046486072b3f30038a640e97711054d8701548a3593bb484e3f726ce" exitCode=0 Oct 31 02:56:40.575220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575231 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="7c3e5603109c93a379725a1085607436bdd7205af407a5c5fa75dcaee8906a95" exitCode=0 Oct 31 02:56:40.575601 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575263 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"cc0b25ff07ef54e1b1774a0bb4bcf7cc7382d1b13130eae9a0c587bb20074752"} Oct 31 02:56:40.575601 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575344 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"b442c59a3b4642c21c701395a10f8156d744a364c4f5e9aa3037e91331dacfac"} Oct 31 02:56:40.575601 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575387 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"2cb0f579826551f4c200bb054ba6b131428472a9b6d0a6bae5b695c2e7bc12da"} Oct 31 02:56:40.575601 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575438 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"f209a5f5046486072b3f30038a640e97711054d8701548a3593bb484e3f726ce"} Oct 31 02:56:40.575601 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:40.575485 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"7c3e5603109c93a379725a1085607436bdd7205af407a5c5fa75dcaee8906a95"} Oct 31 02:56:41.583097 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.583067 4556 generic.go:356] "Generic (PLEG): container finished" podID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerID="ee5fc2aeebfe98f4b8623947e966cac451df07d00c2a6ca9e185cd0ef21986cd" exitCode=0 Oct 31 02:56:41.583528 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.583180 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"ee5fc2aeebfe98f4b8623947e966cac451df07d00c2a6ca9e185cd0ef21986cd"} Oct 31 02:56:41.653563 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.653533 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:41.735330 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735299 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735346 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735381 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735428 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735467 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735500 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735530 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735527 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735570 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735621 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735649 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735684 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735712 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735756 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735787 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.735849 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735821 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735853 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735894 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735923 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735951 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735972 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.736253 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.735990 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\") pod \"57708ca1-bb6c-4b23-aaff-988c38f99d03\" (UID: \"57708ca1-bb6c-4b23-aaff-988c38f99d03\") " Oct 31 02:56:41.736590 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.736325 4556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.736590 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.736447 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.737102 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.737073 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:41.737510 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.737481 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.738331 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.738299 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.739087 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.738808 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.739434 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.739296 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.739434 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.739292 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.739434 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.739366 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:56:41.739627 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.739471 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:41.740797 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.740769 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.742068 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742034 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "kube-api-access-kxhvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:56:41.742276 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742213 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.742276 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742235 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.742276 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742221 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:56:41.742276 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742245 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.742515 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.742280 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.744007 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.743973 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.744086 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.744003 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.750798 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.750769 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config" pod "57708ca1-bb6c-4b23-aaff-988c38f99d03" (UID: "57708ca1-bb6c-4b23-aaff-988c38f99d03"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836893 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhvj\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-kube-api-access-kxhvj\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836929 4556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-kube-rbac-proxy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836939 4556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-grpc-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836949 4556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57708ca1-bb6c-4b23-aaff-988c38f99d03-tls-assets\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836958 4556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836969 4556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836978 4556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-config-out\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836988 4556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.836997 4556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.836996 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837005 4556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837014 4556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837023 4556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837032 4556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-secret-metrics-client-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837041 4556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837050 4556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-config\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837058 4556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-db\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837088 4556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57708ca1-bb6c-4b23-aaff-988c38f99d03-web-config\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837097 4556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-configmap-metrics-client-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:41.837403 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:41.837107 4556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57708ca1-bb6c-4b23-aaff-988c38f99d03-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:56:42.589137 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.589101 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57708ca1-bb6c-4b23-aaff-988c38f99d03","Type":"ContainerDied","Data":"196e9a49db9fe62a829c990b066efa7da16bd523d18036076cddc392fd1d4aa1"} Oct 31 02:56:42.589543 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.589155 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.589543 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.589194 4556 scope.go:122] "RemoveContainer" containerID="cc0b25ff07ef54e1b1774a0bb4bcf7cc7382d1b13130eae9a0c587bb20074752" Oct 31 02:56:42.596568 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.596542 4556 scope.go:122] "RemoveContainer" containerID="b442c59a3b4642c21c701395a10f8156d744a364c4f5e9aa3037e91331dacfac" Oct 31 02:56:42.603189 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.603171 4556 scope.go:122] "RemoveContainer" containerID="ee5fc2aeebfe98f4b8623947e966cac451df07d00c2a6ca9e185cd0ef21986cd" Oct 31 02:56:42.607336 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.607313 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:42.611096 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.611076 4556 scope.go:122] "RemoveContainer" containerID="2cb0f579826551f4c200bb054ba6b131428472a9b6d0a6bae5b695c2e7bc12da" Oct 31 02:56:42.611564 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.611543 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:42.618784 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.618764 4556 scope.go:122] "RemoveContainer" containerID="f209a5f5046486072b3f30038a640e97711054d8701548a3593bb484e3f726ce" Oct 31 02:56:42.625378 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.625356 4556 scope.go:122] "RemoveContainer" containerID="7c3e5603109c93a379725a1085607436bdd7205af407a5c5fa75dcaee8906a95" Oct 31 02:56:42.632885 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.632864 4556 scope.go:122] "RemoveContainer" containerID="df09b50ba2842d960b3779d192fd0e2a799e6d8aa64982899a24388461b87373" Oct 31 02:56:42.640891 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.640864 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:42.641141 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641122 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="prometheus" Oct 31 02:56:42.641141 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641141 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="prometheus" Oct 31 02:56:42.641141 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641153 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641161 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641173 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-thanos" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641181 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-thanos" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641195 4556 cpu_manager.go:414] "containerMap: removing container" podUID="503a089d-2343-450d-bba6-486321ca52e4" containerName="insights-gathering" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641203 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503a089d-2343-450d-bba6-486321ca52e4" containerName="insights-gathering" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641219 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-web" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641226 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-web" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641242 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="config-reloader" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641249 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="config-reloader" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641265 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="init-config-reloader" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641271 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="init-config-reloader" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641280 4556 cpu_manager.go:414] "containerMap: removing container" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="thanos-sidecar" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641287 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="thanos-sidecar" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641342 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="thanos-sidecar" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641350 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="prometheus" Oct 31 02:56:42.641349 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641355 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy" Oct 31 02:56:42.642002 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641360 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-thanos" Oct 31 02:56:42.642002 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641366 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="503a089d-2343-450d-bba6-486321ca52e4" containerName="insights-gathering" Oct 31 02:56:42.642002 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641372 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="kube-rbac-proxy-web" Oct 31 02:56:42.642002 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.641378 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" containerName="config-reloader" Oct 31 02:56:42.646515 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.646498 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.649904 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.649883 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 02:56:42.649904 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.649899 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 02:56:42.650074 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650057 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 02:56:42.650123 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650091 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 02:56:42.650123 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650108 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 02:56:42.650202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650185 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 02:56:42.650202 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650186 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 02:56:42.650268 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650214 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 02:56:42.650268 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650255 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 02:56:42.650331 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650297 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vqvgz\"" Oct 31 02:56:42.650331 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650319 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 02:56:42.650394 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650320 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pagvvcqbttfc\"" Oct 31 02:56:42.650535 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650521 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 02:56:42.650619 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.650605 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 02:56:42.652621 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.652599 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 02:56:42.652709 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.652672 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 02:56:42.664792 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.664757 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:42.744654 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744607 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744654 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744654 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744871 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744675 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744871 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744764 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744871 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744807 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-config-out\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744871 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744839 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.744871 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744866 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745020 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744885 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpjd\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-kube-api-access-tfpjd\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745020 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744956 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745020 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.744987 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745020 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745005 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745137 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745020 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745137 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745061 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745198 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745135 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745198 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745155 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-web-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745198 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745171 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745288 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745197 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745288 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745243 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745351 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745292 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.745351 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.745308 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846025 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.845950 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846025 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.845990 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846025 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846011 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846036 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846068 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846093 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846125 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-config-out\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846183 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846173 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846195 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846221 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpjd\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-kube-api-access-tfpjd\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846265 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846294 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846316 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846337 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846366 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846399 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846449 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-web-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846910 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846472 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846910 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846508 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846910 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846538 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846910 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846774 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.846910 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.846850 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.847168 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.847145 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849592 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849236 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849592 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849504 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849592 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849532 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849592 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849540 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849881 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849607 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14b1a193-5933-4c75-a294-d15a7f16832f-config-out\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.849881 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.849673 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-web-config\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850313 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850013 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850313 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850109 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850313 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850278 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850548 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850339 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850606 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850571 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.850710 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.850686 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.852004 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.851980 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.852087 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.852074 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.852451 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.852431 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14b1a193-5933-4c75-a294-d15a7f16832f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.853258 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.853239 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14b1a193-5933-4c75-a294-d15a7f16832f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.862057 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.862029 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpjd\" (UniqueName: \"kubernetes.io/projected/14b1a193-5933-4c75-a294-d15a7f16832f-kube-api-access-tfpjd\") pod \"prometheus-k8s-0\" (UID: \"14b1a193-5933-4c75-a294-d15a7f16832f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:42.973464 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:42.973398 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:56:43.255448 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:43.255393 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 02:56:43.259056 ip-10-0-139-52 kubenswrapper[4556]: W1031 02:56:43.259023 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b1a193_5933_4c75_a294_d15a7f16832f.slice/crio-11ebee711a5333fe22bf63d63a8ebcc9b876dd44fff4a22c147b565719d2d0c5 WatchSource:0}: Error finding container 11ebee711a5333fe22bf63d63a8ebcc9b876dd44fff4a22c147b565719d2d0c5: Status 404 returned error can't find the container with id 11ebee711a5333fe22bf63d63a8ebcc9b876dd44fff4a22c147b565719d2d0c5 Oct 31 02:56:43.595069 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:43.594981 4556 generic.go:356] "Generic (PLEG): container finished" podID="14b1a193-5933-4c75-a294-d15a7f16832f" containerID="d3190797f95ff01ecd352c0edb73548c9556560821f47a9a6a7f792cef2292af" exitCode=0 Oct 31 02:56:43.595069 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:43.595060 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerDied","Data":"d3190797f95ff01ecd352c0edb73548c9556560821f47a9a6a7f792cef2292af"} Oct 31 02:56:43.595488 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:43.595129 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"11ebee711a5333fe22bf63d63a8ebcc9b876dd44fff4a22c147b565719d2d0c5"} Oct 31 02:56:44.255141 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.255106 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57708ca1-bb6c-4b23-aaff-988c38f99d03" path="/var/lib/kubelet/pods/57708ca1-bb6c-4b23-aaff-988c38f99d03/volumes" Oct 31 02:56:44.601401 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601313 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"251e0fda7380413977e02df53cad040eed4d7dad91cffd33da8985bcc25326a6"} Oct 31 02:56:44.601401 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601396 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"80dfbd809ad3678fb49d9cd073838b51cf7c2d42c8700b78663dffd150f456d9"} Oct 31 02:56:44.601819 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601464 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"8821e567ace564a1938b1b697eda0aa0986e58a13be1566f543a674ad84e66b6"} Oct 31 02:56:44.601819 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601502 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"be34c131f75e99bb389fed492f0556fe062440bd50cae9db4feaf3b8cc2306e5"} Oct 31 02:56:44.601819 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601538 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"de8a8b30219243e73c909b4570f95edf69c768a9aa42064b6a51fe651540815d"} Oct 31 02:56:44.601819 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.601571 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14b1a193-5933-4c75-a294-d15a7f16832f","Type":"ContainerStarted","Data":"e175a5665ebce48e54de6862d2798ab70b2c266a7a73ba4789a80c5fabbcc274"} Oct 31 02:56:44.633798 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:44.633737 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.613119612 podStartE2EDuration="2.633722108s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.602496ms" isStatefulPod=false podCreationTimestamp="2026-10-31 02:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:56:44.632798114 +0000 UTC m=+82.487815266" watchObservedRunningTime="2026-10-31 02:56:44.633722108 +0000 UTC m=+82.488739251" Oct 31 02:56:45.125890 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:45.125857 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:45.255942 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:45.255907 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:45.496126 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:45.496095 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:45.631506 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:45.631472 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:56:45.934504 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:56:45.934459 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:56:47.974115 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:56:47.974080 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:57:07.005887 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:07.005851 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:08.035670 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:08.035633 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:15.934678 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:57:15.934638 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:57:42.974045 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:42.974004 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:57:42.989348 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:42.989321 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:57:43.838262 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:43.838231 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 02:57:45.934735 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:57:45.934693 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:57:50.409679 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:50.409640 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:50.535796 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:50.535768 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:52.070069 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:52.070029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:52.194738 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:52.194700 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:52.436652 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:52.436617 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:57:52.567277 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:57:52.567244 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:03.115702 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:03.115669 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:03.249216 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:03.249186 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:04.677765 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:04.677730 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:04.811367 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:04.811339 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:07.048466 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:07.048405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:08.082104 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:08.082063 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:09.722044 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:09.722003 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:09.861682 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:09.861648 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:14.335549 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:14.335509 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:15.933520 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:58:15.933486 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:58:38.353276 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:38.353240 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:42.622583 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:42.622548 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:42.750546 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:42.750517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:58:45.934261 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:58:45.934217 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:58:48.845590 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:58:48.845562 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:59:07.104106 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:59:07.104018 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:59:08.134220 ip-10-0-139-52 kubenswrapper[4556]: I1031 02:59:08.134187 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 02:59:15.933749 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:59:15.933711 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 02:59:45.934361 ip-10-0-139-52 kubenswrapper[4556]: E1031 02:59:45.934317 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:00:07.209166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:07.209129 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:09.186948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:09.186914 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:09.319854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:09.319825 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:10.477390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:10.477352 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:10.622351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:10.622320 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:11.745373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:11.745338 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:15.934473 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:00:15.934379 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:00:15.980900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:15.980869 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:16.110370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.110334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:16.263611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.263520 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9"] Oct 31 03:00:16.273231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.273202 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.281012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.280989 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9"] Oct 31 03:00:16.282381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.282315 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:16.299368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.299342 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:00:16.299518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.299486 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:00:16.299518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.299499 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-8gmmt\"" Oct 31 03:00:16.299606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.299535 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:00:16.359652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359619 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23938e29-580a-4626-be9a-8218e14034c9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359659 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23938e29-580a-4626-be9a-8218e14034c9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359679 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26s79\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-kube-api-access-26s79\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359801 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359844 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359869 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359888 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.359931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359926 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.360034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.359966 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.411645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.411610 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:16.461240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461209 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461246 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461279 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461448 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461502 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461519 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461563 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23938e29-580a-4626-be9a-8218e14034c9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461596 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23938e29-580a-4626-be9a-8218e14034c9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461621 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26s79\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-kube-api-access-26s79\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461671 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461688 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461831 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.461962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.461929 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.462262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.462242 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23938e29-580a-4626-be9a-8218e14034c9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.463925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.463899 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23938e29-580a-4626-be9a-8218e14034c9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.464117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.464099 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23938e29-580a-4626-be9a-8218e14034c9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.472268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.472233 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26s79\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-kube-api-access-26s79\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.472367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.472300 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23938e29-580a-4626-be9a-8218e14034c9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9\" (UID: \"23938e29-580a-4626-be9a-8218e14034c9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.587306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.587231 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:16.876675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:16.876602 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9"] Oct 31 03:00:16.879604 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:00:16.879575 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23938e29_580a_4626_be9a_8218e14034c9.slice/crio-468d48e0031155a339f7b965c84ba7af4d7fa96f9c40b2dc0184fe8a43393be1 WatchSource:0}: Error finding container 468d48e0031155a339f7b965c84ba7af4d7fa96f9c40b2dc0184fe8a43393be1: Status 404 returned error can't find the container with id 468d48e0031155a339f7b965c84ba7af4d7fa96f9c40b2dc0184fe8a43393be1 Oct 31 03:00:17.289215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:17.289178 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" event={"ID":"23938e29-580a-4626-be9a-8218e14034c9","Type":"ContainerStarted","Data":"468d48e0031155a339f7b965c84ba7af4d7fa96f9c40b2dc0184fe8a43393be1"} Oct 31 03:00:18.943809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:18.943771 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:20.457102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:20.457068 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:00:20.457442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:20.457134 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:00:20.457442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:20.457162 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:00:21.303644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:21.303600 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" event={"ID":"23938e29-580a-4626-be9a-8218e14034c9","Type":"ContainerStarted","Data":"eba071a55282da75b75fd8cfdfdfb5525887b0f69d2815f9677c42d064b55189"} Oct 31 03:00:21.324805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:21.324759 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" podStartSLOduration=1.749829959 podStartE2EDuration="5.324746358s" totalImagesPullingTime="3.574916399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:00:21.324377364 +0000 UTC m=+299.179394543" watchObservedRunningTime="2026-10-31 03:00:21.324746358 +0000 UTC m=+299.179763499" Oct 31 03:00:21.588232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:21.588139 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:21.592589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:21.592563 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:22.307106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:22.307066 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:22.307985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:22.307964 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9" Oct 31 03:00:45.934935 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:00:45.934898 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:00:52.524510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:52.524469 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:55.467536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:55.467501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:55.596153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:55.596119 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:58.776085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:58.776053 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:58.906278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:58.906247 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:59.373684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.373648 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:59.744915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.744859 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:59.896880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.896842 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq"] Oct 31 03:00:59.903727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.903706 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:00:59.948700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.948673 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:00:59.960246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.960184 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq"] Oct 31 03:00:59.972081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972053 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:00:59.972247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972133 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:00:59.972247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972154 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-xc62g\"" Oct 31 03:00:59.972247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972176 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:00:59.972247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972187 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:00:59.972247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:00:59.972239 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:01:00.000723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000656 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.000723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000688 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.000723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000714 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88c8w\" (UniqueName: \"kubernetes.io/projected/d82faf3e-0d6b-4118-9949-611b32936f28-kube-api-access-88c8w\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.000971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.000971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000879 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.000971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.000900 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102217 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102253 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102285 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88c8w\" (UniqueName: \"kubernetes.io/projected/d82faf3e-0d6b-4118-9949-611b32936f28-kube-api-access-88c8w\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102330 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102367 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102391 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.102896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.102871 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-manifests\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.104914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.104885 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.105011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.104927 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-metrics-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.105011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.104956 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d82faf3e-0d6b-4118-9949-611b32936f28-tmp\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.105164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.105147 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d82faf3e-0d6b-4118-9949-611b32936f28-webhook-cert\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.113640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.113615 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88c8w\" (UniqueName: \"kubernetes.io/projected/d82faf3e-0d6b-4118-9949-611b32936f28-kube-api-access-88c8w\") pod \"opendatahub-operator-controller-manager-57b855755-jfdnq\" (UID: \"d82faf3e-0d6b-4118-9949-611b32936f28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.121704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.121671 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:00.217592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.217554 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:00.247955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.247929 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:00.377041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.377012 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:00.508359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.508328 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:00.508539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.508377 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq"] Oct 31 03:01:00.511541 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:00.511515 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82faf3e_0d6b_4118_9949_611b32936f28.slice/crio-8805bf8f9c6675c2bd72eb27de1b891226fecaa7a7bab679fa71da05b4f1bd9f WatchSource:0}: Error finding container 8805bf8f9c6675c2bd72eb27de1b891226fecaa7a7bab679fa71da05b4f1bd9f: Status 404 returned error can't find the container with id 8805bf8f9c6675c2bd72eb27de1b891226fecaa7a7bab679fa71da05b4f1bd9f Oct 31 03:01:00.513149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:00.513131 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:01:01.434751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:01.434715 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" event={"ID":"d82faf3e-0d6b-4118-9949-611b32936f28","Type":"ContainerStarted","Data":"8805bf8f9c6675c2bd72eb27de1b891226fecaa7a7bab679fa71da05b4f1bd9f"} Oct 31 03:01:02.801818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:02.801776 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:02.926497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:02.926456 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:04.447687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:04.447641 4556 generic.go:356] "Generic (PLEG): container finished" podID="d82faf3e-0d6b-4118-9949-611b32936f28" containerID="1c7c9f98d79f6b0f167e69e28fad349f3e57f1bf4aad4476f52611d934670727" exitCode=0 Oct 31 03:01:04.448052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:04.447704 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" event={"ID":"d82faf3e-0d6b-4118-9949-611b32936f28","Type":"ContainerDied","Data":"1c7c9f98d79f6b0f167e69e28fad349f3e57f1bf4aad4476f52611d934670727"} Oct 31 03:01:04.580969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:04.580938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:05.453046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:05.453013 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" event={"ID":"d82faf3e-0d6b-4118-9949-611b32936f28","Type":"ContainerStarted","Data":"6fe417b105aa32299d0a99d6fd69dce250542da005d5ea69162b5b46b1688d94"} Oct 31 03:01:05.453509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:05.453156 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:05.510975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:05.510928 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" podStartSLOduration=2.156444729 podStartE2EDuration="6.510914133s" totalImagesPullingTime="3.942189175s" totalInitContainerRuntime="412.280229ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:00:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:05.509342133 +0000 UTC m=+343.364359275" watchObservedRunningTime="2026-10-31 03:01:05.510914133 +0000 UTC m=+343.365931273" Oct 31 03:01:07.025057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:07.025024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:07.222828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:07.222794 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:08.236307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:08.236274 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:08.625129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:08.625048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:08.764087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:08.764050 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:09.404402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:09.404370 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:09.536103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:09.536074 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:12.501878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:12.501844 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:12.684850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:12.684770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:13.453815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:13.453779 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:13.596647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:13.596613 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:15.867028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:15.866994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:15.933767 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:01:15.933732 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:01:16.060365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.060326 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:16.458921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.458890 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" Oct 31 03:01:16.907384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.907352 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp"] Oct 31 03:01:16.915587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.915563 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:16.920138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.920107 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qplbl\"" Oct 31 03:01:16.920481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.920459 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 03:01:16.920607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.920528 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 03:01:16.921289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:16.921267 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp"] Oct 31 03:01:17.046393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.046356 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.046582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.046502 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ltj\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-kube-api-access-h7ltj\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.147759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.147716 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ltj\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-kube-api-access-h7ltj\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.147954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.147801 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.159703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.159631 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.159824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.159773 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ltj\" (UniqueName: \"kubernetes.io/projected/24ad7cb7-ec84-4441-9739-411ad2e8feb5-kube-api-access-h7ltj\") pod \"cert-manager-cainjector-5c57bbd4fc-4s2fp\" (UID: \"24ad7cb7-ec84-4441-9739-411ad2e8feb5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.237009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.236977 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" Oct 31 03:01:17.510448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:17.510345 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp"] Oct 31 03:01:17.513659 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:17.513632 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ad7cb7_ec84_4441_9739_411ad2e8feb5.slice/crio-7aca43088ef23018fe3daa2a620cb62b83a54cc28f516ebaaa820b77a2c0ff17 WatchSource:0}: Error finding container 7aca43088ef23018fe3daa2a620cb62b83a54cc28f516ebaaa820b77a2c0ff17: Status 404 returned error can't find the container with id 7aca43088ef23018fe3daa2a620cb62b83a54cc28f516ebaaa820b77a2c0ff17 Oct 31 03:01:18.497890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:18.497858 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" event={"ID":"24ad7cb7-ec84-4441-9739-411ad2e8feb5","Type":"ContainerStarted","Data":"7aca43088ef23018fe3daa2a620cb62b83a54cc28f516ebaaa820b77a2c0ff17"} Oct 31 03:01:21.368897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:21.368869 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:21.514445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:21.513316 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" event={"ID":"24ad7cb7-ec84-4441-9739-411ad2e8feb5","Type":"ContainerStarted","Data":"6920cc84fbfa4b7d0db2cc264d51ea603db92d53fd47fc89cfcd37014242236e"} Oct 31 03:01:22.367630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:22.367584 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:22.496638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:22.496596 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:23.379807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:23.379767 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:23.569973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:23.569926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:28.171360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:28.171325 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:28.772200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:28.772163 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:29.640688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:29.640637 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:29.770742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:29.770708 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:31.248900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.248847 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4s2fp" podStartSLOduration=11.39905131 podStartE2EDuration="15.248833673s" totalImagesPullingTime="3.849782363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:21.537277441 +0000 UTC m=+359.392294582" watchObservedRunningTime="2026-10-31 03:01:31.248833673 +0000 UTC m=+369.103850813" Oct 31 03:01:31.249392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.249373 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2"] Oct 31 03:01:31.254123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.254093 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:31.263045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.263021 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:01:31.263177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.263054 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:01:31.263364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.263340 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4px9m\"" Oct 31 03:01:31.264502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.264481 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2"] Oct 31 03:01:31.369006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.368975 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9mw\" (UniqueName: \"kubernetes.io/projected/8a5570e2-5a95-49d3-851c-7cedfdfe0bbc-kube-api-access-5q9mw\") pod \"authorino-operator-7b7bb4bc77-8ljq2\" (UID: \"8a5570e2-5a95-49d3-851c-7cedfdfe0bbc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:31.470033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.469997 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9mw\" (UniqueName: \"kubernetes.io/projected/8a5570e2-5a95-49d3-851c-7cedfdfe0bbc-kube-api-access-5q9mw\") pod \"authorino-operator-7b7bb4bc77-8ljq2\" (UID: \"8a5570e2-5a95-49d3-851c-7cedfdfe0bbc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:31.481537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.481512 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9mw\" (UniqueName: \"kubernetes.io/projected/8a5570e2-5a95-49d3-851c-7cedfdfe0bbc-kube-api-access-5q9mw\") pod \"authorino-operator-7b7bb4bc77-8ljq2\" (UID: \"8a5570e2-5a95-49d3-851c-7cedfdfe0bbc\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:31.566809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.566726 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:31.841293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:31.841219 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2"] Oct 31 03:01:31.846072 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:31.846029 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5570e2_5a95_49d3_851c_7cedfdfe0bbc.slice/crio-a82cb806244dcbf7a28a06174c0e0397427f7af7e45cbfa4990920f0b06645db WatchSource:0}: Error finding container a82cb806244dcbf7a28a06174c0e0397427f7af7e45cbfa4990920f0b06645db: Status 404 returned error can't find the container with id a82cb806244dcbf7a28a06174c0e0397427f7af7e45cbfa4990920f0b06645db Oct 31 03:01:32.030912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.030856 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.182729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.182687 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.245603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.245569 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w"] Oct 31 03:01:32.250269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.250244 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.261688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.261663 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w"] Oct 31 03:01:32.263510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.263490 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:01:32.263753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.263735 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:01:32.263753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.263762 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skvb5\"" Oct 31 03:01:32.277156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.277117 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d839699b-3e0b-419d-bc00-8e07fe5c997a-tmp\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.277322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.277221 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d839699b-3e0b-419d-bc00-8e07fe5c997a-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.277371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.277325 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6ps\" (UniqueName: \"kubernetes.io/projected/d839699b-3e0b-419d-bc00-8e07fe5c997a-kube-api-access-zk6ps\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.355091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.355054 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.378679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.378646 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6ps\" (UniqueName: \"kubernetes.io/projected/d839699b-3e0b-419d-bc00-8e07fe5c997a-kube-api-access-zk6ps\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.379199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.378698 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d839699b-3e0b-419d-bc00-8e07fe5c997a-tmp\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.379199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.378725 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d839699b-3e0b-419d-bc00-8e07fe5c997a-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.381944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.381918 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d839699b-3e0b-419d-bc00-8e07fe5c997a-tls-certs\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.383386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.383359 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d839699b-3e0b-419d-bc00-8e07fe5c997a-tmp\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.397092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.395588 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6ps\" (UniqueName: \"kubernetes.io/projected/d839699b-3e0b-419d-bc00-8e07fe5c997a-kube-api-access-zk6ps\") pod \"kube-auth-proxy-5d9fdc54f-dx65w\" (UID: \"d839699b-3e0b-419d-bc00-8e07fe5c997a\") " pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.398641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.397637 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf"] Oct 31 03:01:32.401004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.400984 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.412392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.412358 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jxj87\"" Oct 31 03:01:32.418439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.418401 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf"] Oct 31 03:01:32.480095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480018 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480062 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480083 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480219 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480250 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bqb\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-kube-api-access-c6bqb\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480305 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480408 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.480521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.480452 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.507392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.507342 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.550193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.550161 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" event={"ID":"8a5570e2-5a95-49d3-851c-7cedfdfe0bbc","Type":"ContainerStarted","Data":"a82cb806244dcbf7a28a06174c0e0397427f7af7e45cbfa4990920f0b06645db"} Oct 31 03:01:32.565308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.565284 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" Oct 31 03:01:32.581797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.581763 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.581924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.581815 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.581924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.581864 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582015 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582174 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582398 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582505 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582532 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582562 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582627 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bqb\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-kube-api-access-c6bqb\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582443 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.582904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.582218 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.583215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.583191 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.583792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.583705 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.585117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.585093 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.585437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.585394 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.593834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.593807 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.594932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.594906 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bqb\" (UniqueName: \"kubernetes.io/projected/cbb6aa56-2c35-45a7-b37d-98ec927dd0ee-kube-api-access-c6bqb\") pod \"data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf\" (UID: \"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.651643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.651606 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.717952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.717925 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:32.866760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.866722 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:32.869284 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:32.869257 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd839699b_3e0b_419d_bc00_8e07fe5c997a.slice/crio-e306d2a1d969946ca30b8b288affa531f003241788967fe46178a60ead718e55 WatchSource:0}: Error finding container e306d2a1d969946ca30b8b288affa531f003241788967fe46178a60ead718e55: Status 404 returned error can't find the container with id e306d2a1d969946ca30b8b288affa531f003241788967fe46178a60ead718e55 Oct 31 03:01:32.870112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:32.870092 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w"] Oct 31 03:01:33.002574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.002541 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf"] Oct 31 03:01:33.006646 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:33.006608 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb6aa56_2c35_45a7_b37d_98ec927dd0ee.slice/crio-1d1648182c0a170d51b21a003f6734abff22919b4fae7c3d41e1273735d23fdd WatchSource:0}: Error finding container 1d1648182c0a170d51b21a003f6734abff22919b4fae7c3d41e1273735d23fdd: Status 404 returned error can't find the container with id 1d1648182c0a170d51b21a003f6734abff22919b4fae7c3d41e1273735d23fdd Oct 31 03:01:33.008757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.008729 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:33.008966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.008940 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:01:33.009027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.009010 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:01:33.009062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.009039 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:01:33.558358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.557744 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" event={"ID":"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee","Type":"ContainerStarted","Data":"cadd5a41952a928bfee8d454b601babfad61aa0454bb8a7e63af4b9350f58c01"} Oct 31 03:01:33.558358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.557843 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" event={"ID":"cbb6aa56-2c35-45a7-b37d-98ec927dd0ee","Type":"ContainerStarted","Data":"1d1648182c0a170d51b21a003f6734abff22919b4fae7c3d41e1273735d23fdd"} Oct 31 03:01:33.560697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.560654 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" event={"ID":"d839699b-3e0b-419d-bc00-8e07fe5c997a","Type":"ContainerStarted","Data":"e306d2a1d969946ca30b8b288affa531f003241788967fe46178a60ead718e55"} Oct 31 03:01:33.583223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.583158 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" podStartSLOduration=1.583140159 podStartE2EDuration="1.583140159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:33.581072399 +0000 UTC m=+371.436089552" watchObservedRunningTime="2026-10-31 03:01:33.583140159 +0000 UTC m=+371.438157301" Oct 31 03:01:33.718946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:33.718856 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:34.021834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:34.021797 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:34.532477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:34.532435 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:34.675274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:34.675238 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:34.724294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:34.724262 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:35.185190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:35.185138 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:35.570101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:35.570074 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:35.571219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:35.571197 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf" Oct 31 03:01:35.846071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:35.845985 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:36.490751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.490713 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h"] Oct 31 03:01:36.493942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.493923 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.499656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.499606 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:01:36.499803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.499789 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:01:36.501188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.501163 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 03:01:36.507132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.507112 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h"] Oct 31 03:01:36.563044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.563011 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-47m96\"" Oct 31 03:01:36.574996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.574962 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" event={"ID":"8a5570e2-5a95-49d3-851c-7cedfdfe0bbc","Type":"ContainerStarted","Data":"c35d08a8e9da6a513924119ca755318e219f253bed5970f32964338351cc1c99"} Oct 31 03:01:36.575104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.575080 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:36.576627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.576604 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" event={"ID":"d839699b-3e0b-419d-bc00-8e07fe5c997a","Type":"ContainerStarted","Data":"b7511f5e8ef4cb119b2b4583de34fa4de7af3a1d3681f10d1fabbcef823c69c8"} Oct 31 03:01:36.593132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.593087 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" podStartSLOduration=1.906874298 podStartE2EDuration="5.593074813s" totalImagesPullingTime="3.686200515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:36.592431976 +0000 UTC m=+374.447449117" watchObservedRunningTime="2026-10-31 03:01:36.593074813 +0000 UTC m=+374.448091948" Oct 31 03:01:36.614428 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.614372 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d9fdc54f-dx65w" podStartSLOduration=1.550870516 podStartE2EDuration="4.614358176s" totalImagesPullingTime="3.06348766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:36.613108842 +0000 UTC m=+374.468125982" watchObservedRunningTime="2026-10-31 03:01:36.614358176 +0000 UTC m=+374.469375318" Oct 31 03:01:36.628084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.628024 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.628233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.628208 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f209d2-7d1a-4b6d-8e04-e50fc1427597-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.628339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.628314 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpnh\" (UniqueName: \"kubernetes.io/projected/84f209d2-7d1a-4b6d-8e04-e50fc1427597-kube-api-access-vhpnh\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.628717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.628686 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.729980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.729948 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.729980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.729991 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.730225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.730063 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f209d2-7d1a-4b6d-8e04-e50fc1427597-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.730225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.730097 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpnh\" (UniqueName: \"kubernetes.io/projected/84f209d2-7d1a-4b6d-8e04-e50fc1427597-kube-api-access-vhpnh\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.730544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.730509 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.730859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.730817 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84f209d2-7d1a-4b6d-8e04-e50fc1427597-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.731734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.731715 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f209d2-7d1a-4b6d-8e04-e50fc1427597-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.748267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.748204 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpnh\" (UniqueName: \"kubernetes.io/projected/84f209d2-7d1a-4b6d-8e04-e50fc1427597-kube-api-access-vhpnh\") pod \"kserve-module-controller-manager-7566cf5fc8-mmt5h\" (UID: \"84f209d2-7d1a-4b6d-8e04-e50fc1427597\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:36.807218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:36.807185 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:37.087727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:37.087586 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h"] Oct 31 03:01:37.091619 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:01:37.091593 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f209d2_7d1a_4b6d_8e04_e50fc1427597.slice/crio-55aa679d285ae6cf813ca2220d444e1d033543d8ecd138b682bafc70e5c56784 WatchSource:0}: Error finding container 55aa679d285ae6cf813ca2220d444e1d033543d8ecd138b682bafc70e5c56784: Status 404 returned error can't find the container with id 55aa679d285ae6cf813ca2220d444e1d033543d8ecd138b682bafc70e5c56784 Oct 31 03:01:37.582474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:37.582437 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" event={"ID":"84f209d2-7d1a-4b6d-8e04-e50fc1427597","Type":"ContainerStarted","Data":"55aa679d285ae6cf813ca2220d444e1d033543d8ecd138b682bafc70e5c56784"} Oct 31 03:01:38.820647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:38.820601 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:38.947566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:38.947540 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:39.591365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:39.591330 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" event={"ID":"84f209d2-7d1a-4b6d-8e04-e50fc1427597","Type":"ContainerStarted","Data":"621625eb3a4e7673301b574e908b6dcffcb2f03f6c847ad8db6861cb3eb8cc67"} Oct 31 03:01:39.591565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:39.591466 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:01:39.610809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:39.610754 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" podStartSLOduration=1.782390329 podStartE2EDuration="3.610741836s" totalImagesPullingTime="1.828351507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:01:39.609653791 +0000 UTC m=+377.464670931" watchObservedRunningTime="2026-10-31 03:01:39.610741836 +0000 UTC m=+377.465758981" Oct 31 03:01:40.176319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:40.176286 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:40.305954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:40.305916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:40.576434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:40.576345 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:40.703273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:40.703240 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:40.976399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:40.976361 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:41.101393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:41.101360 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:45.934267 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:01:45.934225 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:01:47.585543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:47.585512 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8ljq2" Oct 31 03:01:48.807460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:48.807403 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:48.937467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:48.937434 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:51.281213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:51.281180 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:51.412231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:51.412197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:51.586620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:51.586544 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:51.721292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:51.721259 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:53.046191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:53.046157 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:56.068253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:56.068219 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:56.201386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:56.201356 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:59.020641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.020611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:59.148805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.148770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:59.667397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.667365 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:01:59.673065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.673042 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:01:59.678540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.678509 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:01:59.680873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.680833 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-77b77\"" Oct 31 03:01:59.763431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.763380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:59.816794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.816759 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:01:59.821338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.821306 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:01:59.830759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.830731 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:01:59.834319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.834292 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlv25\" (UniqueName: \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\") pod \"authorino-57f7f8659f-zbpwk\" (UID: \"af2e9da7-777c-43b8-981e-650e61c5f739\") " pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:01:59.887991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.887957 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:01:59.934885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.934808 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlv25\" (UniqueName: \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\") pod \"authorino-57f7f8659f-zbpwk\" (UID: \"af2e9da7-777c-43b8-981e-650e61c5f739\") " pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:01:59.934885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.934867 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn28t\" (UniqueName: \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\") pod \"authorino-56c6c9ddd5-w8rr4\" (UID: \"a9221f25-c390-4200-ab15-02e4171386f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:01:59.946132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.946105 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlv25\" (UniqueName: \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\") pod \"authorino-57f7f8659f-zbpwk\" (UID: \"af2e9da7-777c-43b8-981e-650e61c5f739\") " pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:01:59.985141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:01:59.985109 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:02:00.036315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.036285 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn28t\" (UniqueName: \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\") pod \"authorino-56c6c9ddd5-w8rr4\" (UID: \"a9221f25-c390-4200-ab15-02e4171386f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:02:00.057447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.057363 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn28t\" (UniqueName: \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\") pod \"authorino-56c6c9ddd5-w8rr4\" (UID: \"a9221f25-c390-4200-ab15-02e4171386f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:02:00.134931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.134895 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:02:00.255513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.255480 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:02:00.255778 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:02:00.255756 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf2e9da7_777c_43b8_981e_650e61c5f739.slice/crio-8e1be5c3ece125bdbbfd0d17aa0d956d9db4245f1f35dec627c7cb7f91eb63fb WatchSource:0}: Error finding container 8e1be5c3ece125bdbbfd0d17aa0d956d9db4245f1f35dec627c7cb7f91eb63fb: Status 404 returned error can't find the container with id 8e1be5c3ece125bdbbfd0d17aa0d956d9db4245f1f35dec627c7cb7f91eb63fb Oct 31 03:02:00.405157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.405131 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:02:00.407620 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:02:00.407593 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9221f25_c390_4200_ab15_02e4171386f8.slice/crio-897838267df156cb3a7e932c05dee7306b4d66f09e1e9ab94c40df9145da9658 WatchSource:0}: Error finding container 897838267df156cb3a7e932c05dee7306b4d66f09e1e9ab94c40df9145da9658: Status 404 returned error can't find the container with id 897838267df156cb3a7e932c05dee7306b4d66f09e1e9ab94c40df9145da9658 Oct 31 03:02:00.596873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.596786 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mmt5h" Oct 31 03:02:00.676593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.676555 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" event={"ID":"af2e9da7-777c-43b8-981e-650e61c5f739","Type":"ContainerStarted","Data":"8e1be5c3ece125bdbbfd0d17aa0d956d9db4245f1f35dec627c7cb7f91eb63fb"} Oct 31 03:02:00.677689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:00.677663 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" event={"ID":"a9221f25-c390-4200-ab15-02e4171386f8","Type":"ContainerStarted","Data":"897838267df156cb3a7e932c05dee7306b4d66f09e1e9ab94c40df9145da9658"} Oct 31 03:02:01.228844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:01.228790 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:01.390295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:01.390261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:01.547602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:01.547517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:01.722528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:01.722481 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:01.872038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:01.871916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:02.015678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:02.015621 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:02.689525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:02.689485 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" event={"ID":"a9221f25-c390-4200-ab15-02e4171386f8","Type":"ContainerStarted","Data":"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70"} Oct 31 03:02:02.707662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:02.707600 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" podStartSLOduration=1.576903711 podStartE2EDuration="3.70758174s" totalImagesPullingTime="2.130678029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:02.706586285 +0000 UTC m=+400.561603425" watchObservedRunningTime="2026-10-31 03:02:02.70758174 +0000 UTC m=+400.562598881" Oct 31 03:02:02.733656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:02.733629 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:02:03.696067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:03.696024 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" event={"ID":"af2e9da7-777c-43b8-981e-650e61c5f739","Type":"ContainerStarted","Data":"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6"} Oct 31 03:02:03.696594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:03.696104 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" podUID="af2e9da7-777c-43b8-981e-650e61c5f739" containerName="authorino" containerID="cri-o://9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6" gracePeriod=30 Oct 31 03:02:03.711295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:03.711251 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" podStartSLOduration=2.245190145 podStartE2EDuration="4.711239019s" totalImagesPullingTime="2.466048874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:03.711010708 +0000 UTC m=+401.566027848" watchObservedRunningTime="2026-10-31 03:02:03.711239019 +0000 UTC m=+401.566256157" Oct 31 03:02:03.931611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:03.931585 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:02:04.074371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.074270 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\" (UniqueName: \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\") pod \"af2e9da7-777c-43b8-981e-650e61c5f739\" (UID: \"af2e9da7-777c-43b8-981e-650e61c5f739\") " Oct 31 03:02:04.076409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.076378 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25" pod "af2e9da7-777c-43b8-981e-650e61c5f739" (UID: "af2e9da7-777c-43b8-981e-650e61c5f739"). InnerVolumeSpecName "kube-api-access-jlv25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:02:04.175585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.175555 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlv25\" (UniqueName: \"kubernetes.io/projected/af2e9da7-777c-43b8-981e-650e61c5f739-kube-api-access-jlv25\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:02:04.702179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.702140 4556 generic.go:356] "Generic (PLEG): container finished" podID="af2e9da7-777c-43b8-981e-650e61c5f739" containerID="9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6" exitCode=0 Oct 31 03:02:04.702684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.702189 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" Oct 31 03:02:04.702684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.702212 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" event={"ID":"af2e9da7-777c-43b8-981e-650e61c5f739","Type":"ContainerDied","Data":"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6"} Oct 31 03:02:04.702684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.702243 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zbpwk" event={"ID":"af2e9da7-777c-43b8-981e-650e61c5f739","Type":"ContainerDied","Data":"8e1be5c3ece125bdbbfd0d17aa0d956d9db4245f1f35dec627c7cb7f91eb63fb"} Oct 31 03:02:04.702684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.702260 4556 scope.go:122] "RemoveContainer" containerID="9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6" Oct 31 03:02:04.710088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.710069 4556 scope.go:122] "RemoveContainer" containerID="9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6" Oct 31 03:02:04.710339 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:02:04.710320 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6\": container with ID starting with 9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6 not found: ID does not exist" containerID="9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6" Oct 31 03:02:04.710399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.710351 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6"} err="failed to get container status \"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6\": rpc error: code = NotFound desc = could not find container \"9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6\": container with ID starting with 9d48e7f9bf509a8aaaa0fa40dd3e0410ea62e16dc8fa769ddb06eec1f0d0f9d6 not found: ID does not exist" Oct 31 03:02:04.717617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.717595 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:02:04.723400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:04.723379 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zbpwk"] Oct 31 03:02:06.254843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:06.254808 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2e9da7-777c-43b8-981e-650e61c5f739" path="/var/lib/kubelet/pods/af2e9da7-777c-43b8-981e-650e61c5f739/volumes" Oct 31 03:02:07.267285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:07.267245 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:08.273362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:08.273315 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:10.216142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:10.216107 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:10.347804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:10.347765 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:12.651769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:12.651736 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:12.781737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:12.781705 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:15.934235 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:02:15.934200 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:02:17.457531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.457501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:17.590270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.590233 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:02:17.590601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.590584 4556 cpu_manager.go:414] "containerMap: removing container" podUID="af2e9da7-777c-43b8-981e-650e61c5f739" containerName="authorino" Oct 31 03:02:17.590601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.590601 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2e9da7-777c-43b8-981e-650e61c5f739" containerName="authorino" Oct 31 03:02:17.590726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.590681 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2e9da7-777c-43b8-981e-650e61c5f739" containerName="authorino" Oct 31 03:02:17.598840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.598817 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.606691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.606668 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tdjqk\"" Oct 31 03:02:17.607677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.607631 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:02:17.611663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.611637 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 03:02:17.614007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.613982 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:17.647825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.647793 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:02:17.686019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.685989 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mg6j\" (UniqueName: \"kubernetes.io/projected/4ddc1e3c-e417-4674-9c14-fa83a173aa02-kube-api-access-2mg6j\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.686166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.686042 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ddc1e3c-e417-4674-9c14-fa83a173aa02-config-file\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.787486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.787384 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mg6j\" (UniqueName: \"kubernetes.io/projected/4ddc1e3c-e417-4674-9c14-fa83a173aa02-kube-api-access-2mg6j\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.787486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.787471 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ddc1e3c-e417-4674-9c14-fa83a173aa02-config-file\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.788064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.788044 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ddc1e3c-e417-4674-9c14-fa83a173aa02-config-file\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.798858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.798825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mg6j\" (UniqueName: \"kubernetes.io/projected/4ddc1e3c-e417-4674-9c14-fa83a173aa02-kube-api-access-2mg6j\") pod \"limitador-limitador-7c9b87d574-cr6cj\" (UID: \"4ddc1e3c-e417-4674-9c14-fa83a173aa02\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:17.913869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:17.913831 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:18.201061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:18.200986 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:02:18.203576 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:02:18.203542 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddc1e3c_e417_4674_9c14_fa83a173aa02.slice/crio-88fa6bbcca11c0a9e5e9616de0594de7ed47aac6098b85145091858fd0bd9ccb WatchSource:0}: Error finding container 88fa6bbcca11c0a9e5e9616de0594de7ed47aac6098b85145091858fd0bd9ccb: Status 404 returned error can't find the container with id 88fa6bbcca11c0a9e5e9616de0594de7ed47aac6098b85145091858fd0bd9ccb Oct 31 03:02:18.756751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:18.756709 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" event={"ID":"4ddc1e3c-e417-4674-9c14-fa83a173aa02","Type":"ContainerStarted","Data":"88fa6bbcca11c0a9e5e9616de0594de7ed47aac6098b85145091858fd0bd9ccb"} Oct 31 03:02:19.764443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:19.764341 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" event={"ID":"4ddc1e3c-e417-4674-9c14-fa83a173aa02","Type":"ContainerStarted","Data":"d12407aa0171942571c50ada08c68905317d6a88dadb72a7c91d2f2b16147f1c"} Oct 31 03:02:19.764878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:19.764465 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:19.782738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:19.782682 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" podStartSLOduration=1.525802949 podStartE2EDuration="2.782665336s" totalImagesPullingTime="1.256862387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:02:19.781737013 +0000 UTC m=+417.636754154" watchObservedRunningTime="2026-10-31 03:02:19.782665336 +0000 UTC m=+417.637682475" Oct 31 03:02:24.930286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:24.930249 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.031853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.031813 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.198231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.198193 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.327608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.327537 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.458008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.457978 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.592572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.592497 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:30.769087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:30.769060 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-cr6cj" Oct 31 03:02:31.746022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:31.745987 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:31.909915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:31.909881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:32.048115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:32.048022 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:36.119998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:36.119965 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:36.263513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:36.263482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:36.946433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:36.946356 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:45.934771 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:02:45.934723 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:02:52.376308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:52.376260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:56.627518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:56.627485 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:56.765589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:56.765556 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:02:56.897553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:02:56.897472 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:00.789578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:00.789534 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:00.916650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:00.916606 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:02.280335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:02.280297 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:02.432993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:02.432960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:07.323951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:07.323915 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:08.314795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:08.314766 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:15.935428 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:03:15.935371 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:03:29.738129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:29.738084 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:29.867040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:29.866999 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:31.083515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:31.083483 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:31.220907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:31.220875 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:31.368269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:31.368201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:32.089952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:32.089908 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:03:34.983023 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:34.982945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:41.197473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.197409 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:41.353408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.353374 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:03:41.493245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.493168 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wnnwh"] Oct 31 03:03:41.497642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.497624 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:41.505244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.505213 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wnnwh"] Oct 31 03:03:41.582841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.582807 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wnnwh"] Oct 31 03:03:41.583023 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:03:41.583004 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dcm5m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" podUID="d7156956-44c1-4e39-bf3e-56d8854e3ca9" Oct 31 03:03:41.611770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.611732 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:41.611929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.611886 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcm5m\" (UniqueName: \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\") pod \"authorino-6f6944bcdd-wnnwh\" (UID: \"d7156956-44c1-4e39-bf3e-56d8854e3ca9\") " pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:41.614888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.614873 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.619274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.619253 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 03:03:41.624162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.624137 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:41.712947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.712915 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.712947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.712951 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bdm\" (UniqueName: \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.713230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.713059 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcm5m\" (UniqueName: \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\") pod \"authorino-6f6944bcdd-wnnwh\" (UID: \"d7156956-44c1-4e39-bf3e-56d8854e3ca9\") " pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:41.726465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.726437 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcm5m\" (UniqueName: \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\") pod \"authorino-6f6944bcdd-wnnwh\" (UID: \"d7156956-44c1-4e39-bf3e-56d8854e3ca9\") " pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:41.813730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.813646 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.813730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.813685 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bdm\" (UniqueName: \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.816063 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.816041 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.825066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.825042 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bdm\" (UniqueName: \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\") pod \"authorino-6889cbd8c4-9dsqt\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.888574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.888545 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:41.888761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.888748 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:41.918571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.918535 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:03:41.922244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.922215 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:41.930593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:41.930566 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:03:42.014808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.014768 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.014955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.014833 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2qw\" (UniqueName: \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.093739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.093660 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:42.100274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.100250 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:42.116109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.116079 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2qw\" (UniqueName: \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.116264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.116151 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.118613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.118588 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.127687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.127665 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2qw\" (UniqueName: \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\") pod \"authorino-7585446f65-hdv7j\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.168011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.167974 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:42.171427 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:03:42.171386 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a7c5d75_b5f5_4356_bd05_ed3a2173cfa3.slice/crio-b37f28a989f26c6d55b5964897921547118226ad4ff9ea4d6a7d146feac37c60 WatchSource:0}: Error finding container b37f28a989f26c6d55b5964897921547118226ad4ff9ea4d6a7d146feac37c60: Status 404 returned error can't find the container with id b37f28a989f26c6d55b5964897921547118226ad4ff9ea4d6a7d146feac37c60 Oct 31 03:03:42.217255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.217216 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\" (UniqueName: \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\") pod \"d7156956-44c1-4e39-bf3e-56d8854e3ca9\" (UID: \"d7156956-44c1-4e39-bf3e-56d8854e3ca9\") " Oct 31 03:03:42.219255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.219230 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m" pod "d7156956-44c1-4e39-bf3e-56d8854e3ca9" (UID: "d7156956-44c1-4e39-bf3e-56d8854e3ca9"). InnerVolumeSpecName "kube-api-access-dcm5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:03:42.237188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.237164 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:03:42.318267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.318237 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcm5m\" (UniqueName: \"kubernetes.io/projected/d7156956-44c1-4e39-bf3e-56d8854e3ca9-kube-api-access-dcm5m\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:42.507561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:42.507527 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:03:42.510440 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:03:42.510395 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99e777cc_346f_4b0d_9f3a_beae5a07df65.slice/crio-51505d0b77d4c45958dcbc26f0d6427042f9aca87d5ba6af9ad975a5aed15ec3 WatchSource:0}: Error finding container 51505d0b77d4c45958dcbc26f0d6427042f9aca87d5ba6af9ad975a5aed15ec3: Status 404 returned error can't find the container with id 51505d0b77d4c45958dcbc26f0d6427042f9aca87d5ba6af9ad975a5aed15ec3 Oct 31 03:03:43.098349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:43.098318 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7585446f65-hdv7j" event={"ID":"99e777cc-346f-4b0d-9f3a-beae5a07df65","Type":"ContainerStarted","Data":"51505d0b77d4c45958dcbc26f0d6427042f9aca87d5ba6af9ad975a5aed15ec3"} Oct 31 03:03:43.099541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:43.099518 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" event={"ID":"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3","Type":"ContainerStarted","Data":"b37f28a989f26c6d55b5964897921547118226ad4ff9ea4d6a7d146feac37c60"} Oct 31 03:03:43.099642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:43.099554 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wnnwh" Oct 31 03:03:43.124911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:43.124877 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wnnwh"] Oct 31 03:03:43.130896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:43.130865 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wnnwh"] Oct 31 03:03:44.107179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.107141 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" event={"ID":"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3","Type":"ContainerStarted","Data":"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73"} Oct 31 03:03:44.107673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.107190 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" podUID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" containerName="authorino" containerID="cri-o://3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73" gracePeriod=30 Oct 31 03:03:44.108769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.108744 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7585446f65-hdv7j" event={"ID":"99e777cc-346f-4b0d-9f3a-beae5a07df65","Type":"ContainerStarted","Data":"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd"} Oct 31 03:03:44.125658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.125599 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" podStartSLOduration=2.244488862 podStartE2EDuration="3.125580703s" totalImagesPullingTime="881.091841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:44.124462612 +0000 UTC m=+501.979479750" watchObservedRunningTime="2026-10-31 03:03:44.125580703 +0000 UTC m=+501.980597845" Oct 31 03:03:44.140447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.140373 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7585446f65-hdv7j" podStartSLOduration=2.321935419 podStartE2EDuration="3.140361066s" totalImagesPullingTime="818.425647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:03:44.139970603 +0000 UTC m=+501.994987754" watchObservedRunningTime="2026-10-31 03:03:44.140361066 +0000 UTC m=+501.995378206" Oct 31 03:03:44.169802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.169769 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:03:44.170003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.169980 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" podUID="a9221f25-c390-4200-ab15-02e4171386f8" containerName="authorino" containerID="cri-o://e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70" gracePeriod=30 Oct 31 03:03:44.255783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.255752 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7156956-44c1-4e39-bf3e-56d8854e3ca9" path="/var/lib/kubelet/pods/d7156956-44c1-4e39-bf3e-56d8854e3ca9/volumes" Oct 31 03:03:44.432798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.432775 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:44.435465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.435443 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:03:44.538870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.538833 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\" (UniqueName: \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\") pod \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " Oct 31 03:03:44.539043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.538900 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\" (UniqueName: \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\") pod \"a9221f25-c390-4200-ab15-02e4171386f8\" (UID: \"a9221f25-c390-4200-ab15-02e4171386f8\") " Oct 31 03:03:44.539043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.539010 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\" (UniqueName: \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\") pod \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\" (UID: \"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3\") " Oct 31 03:03:44.541134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.541099 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm" pod "7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" (UID: "7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3"). InnerVolumeSpecName "kube-api-access-q2bdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:03:44.541134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.541124 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t" pod "a9221f25-c390-4200-ab15-02e4171386f8" (UID: "a9221f25-c390-4200-ab15-02e4171386f8"). InnerVolumeSpecName "kube-api-access-fn28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:03:44.548582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.548553 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert" pod "7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" (UID: "7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:03:44.640558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.640470 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bdm\" (UniqueName: \"kubernetes.io/projected/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-kube-api-access-q2bdm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:44.640558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.640505 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn28t\" (UniqueName: \"kubernetes.io/projected/a9221f25-c390-4200-ab15-02e4171386f8-kube-api-access-fn28t\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:44.640558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:44.640515 4556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3-tls-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:03:45.113837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.113803 4556 generic.go:356] "Generic (PLEG): container finished" podID="a9221f25-c390-4200-ab15-02e4171386f8" containerID="e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70" exitCode=0 Oct 31 03:03:45.114310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.113846 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" Oct 31 03:03:45.114310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.113847 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" event={"ID":"a9221f25-c390-4200-ab15-02e4171386f8","Type":"ContainerDied","Data":"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70"} Oct 31 03:03:45.114310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.113964 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w8rr4" event={"ID":"a9221f25-c390-4200-ab15-02e4171386f8","Type":"ContainerDied","Data":"897838267df156cb3a7e932c05dee7306b4d66f09e1e9ab94c40df9145da9658"} Oct 31 03:03:45.114310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.113981 4556 scope.go:122] "RemoveContainer" containerID="e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70" Oct 31 03:03:45.115382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.115354 4556 generic.go:356] "Generic (PLEG): container finished" podID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" containerID="3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73" exitCode=0 Oct 31 03:03:45.115521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.115424 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" Oct 31 03:03:45.115521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.115457 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" event={"ID":"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3","Type":"ContainerDied","Data":"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73"} Oct 31 03:03:45.115521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.115489 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-9dsqt" event={"ID":"7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3","Type":"ContainerDied","Data":"b37f28a989f26c6d55b5964897921547118226ad4ff9ea4d6a7d146feac37c60"} Oct 31 03:03:45.122653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.122633 4556 scope.go:122] "RemoveContainer" containerID="e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70" Oct 31 03:03:45.122941 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:03:45.122921 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70\": container with ID starting with e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70 not found: ID does not exist" containerID="e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70" Oct 31 03:03:45.123021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.122949 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70"} err="failed to get container status \"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70\": rpc error: code = NotFound desc = could not find container \"e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70\": container with ID starting with e4bb9903828f7e200d853f617bcc3d16d1d4b51f6bc9e4479ddb707c215daa70 not found: ID does not exist" Oct 31 03:03:45.123021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.122966 4556 scope.go:122] "RemoveContainer" containerID="3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73" Oct 31 03:03:45.132444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.132425 4556 scope.go:122] "RemoveContainer" containerID="3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73" Oct 31 03:03:45.132734 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:03:45.132716 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73\": container with ID starting with 3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73 not found: ID does not exist" containerID="3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73" Oct 31 03:03:45.132788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.132743 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73"} err="failed to get container status \"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73\": rpc error: code = NotFound desc = could not find container \"3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73\": container with ID starting with 3151b04ab5f723aafa67704f453fba5e0006510e190bbf878143df6cb2bb2c73 not found: ID does not exist" Oct 31 03:03:45.138843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.138818 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:03:45.142953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.142930 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w8rr4"] Oct 31 03:03:45.152778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.152753 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:45.155320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:45.155295 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-9dsqt"] Oct 31 03:03:45.934901 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:03:45.934865 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:03:46.255843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:46.255753 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" path="/var/lib/kubelet/pods/7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3/volumes" Oct 31 03:03:46.256333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:03:46.256207 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9221f25-c390-4200-ab15-02e4171386f8" path="/var/lib/kubelet/pods/a9221f25-c390-4200-ab15-02e4171386f8/volumes" Oct 31 03:04:07.374428 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:07.374383 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:08.366383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:08.366350 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:15.933865 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:04:15.933825 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:04:19.810300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:19.810264 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:19.965144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:19.965113 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:30.577110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:30.577072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:30.706958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:30.706926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:38.954218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954172 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd"] Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954577 4556 cpu_manager.go:414] "containerMap: removing container" podUID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" containerName="authorino" Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954596 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" containerName="authorino" Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954620 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a9221f25-c390-4200-ab15-02e4171386f8" containerName="authorino" Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954626 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9221f25-c390-4200-ab15-02e4171386f8" containerName="authorino" Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954690 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7c5d75-b5f5-4356-bd05-ed3a2173cfa3" containerName="authorino" Oct 31 03:04:38.954775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.954701 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9221f25-c390-4200-ab15-02e4171386f8" containerName="authorino" Oct 31 03:04:38.957609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.957586 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:38.960731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.960698 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 03:04:38.960731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.960722 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 03:04:38.960871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.960726 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 03:04:38.960871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.960744 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-88f92\"" Oct 31 03:04:38.968715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:38.968691 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd"] Oct 31 03:04:39.052176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.052140 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:39.119952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.119912 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.120145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.119960 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.120145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.119990 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.120145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.120064 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j699g\" (UniqueName: \"kubernetes.io/projected/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-kube-api-access-j699g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.120274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.120170 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.120274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.120207 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.220916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220830 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.220916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220872 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.220916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220891 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.220916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220911 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j699g\" (UniqueName: \"kubernetes.io/projected/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-kube-api-access-j699g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.221252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220943 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.221252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.220980 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.221357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.221296 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.221403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.221381 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.221471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.221439 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.223303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.223232 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.223688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.223643 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.232400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.232367 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j699g\" (UniqueName: \"kubernetes.io/projected/ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205-kube-api-access-j699g\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd\" (UID: \"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.269719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.269691 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:39.491536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.491456 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:04:39.549009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:39.548897 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd"] Oct 31 03:04:39.552246 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:04:39.552214 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca0e3e54_3fc4_4bd5_9e2f_c61cdec88205.slice/crio-8cc335f753e3fe4bdda0ded7e800735a0c07600d88bcfa509e4466f7b2eb6ca1 WatchSource:0}: Error finding container 8cc335f753e3fe4bdda0ded7e800735a0c07600d88bcfa509e4466f7b2eb6ca1: Status 404 returned error can't find the container with id 8cc335f753e3fe4bdda0ded7e800735a0c07600d88bcfa509e4466f7b2eb6ca1 Oct 31 03:04:40.335344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:40.335307 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" event={"ID":"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205","Type":"ContainerStarted","Data":"8cc335f753e3fe4bdda0ded7e800735a0c07600d88bcfa509e4466f7b2eb6ca1"} Oct 31 03:04:41.080600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:41.080569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:41.340701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:41.340617 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" event={"ID":"ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205","Type":"ContainerStarted","Data":"b454fbf7e091b42de696ec5f7c0882ed06cc86639e3e259649f9ab4b1c4e3d23"} Oct 31 03:04:41.358878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:41.358797 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podStartSLOduration=1.835763133 podStartE2EDuration="3.358781476s" totalImagesPullingTime="1.523018343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:41.357872083 +0000 UTC m=+559.212889224" watchObservedRunningTime="2026-10-31 03:04:41.358781476 +0000 UTC m=+559.213798615" Oct 31 03:04:41.490755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:41.490721 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:41.664199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:41.664162 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:04:42.270323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:42.270286 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:44.071942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.071907 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5"] Oct 31 03:04:44.075585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.075568 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.078603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.078583 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 03:04:44.088592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.088567 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5"] Oct 31 03:04:44.105497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105462 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.105621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105512 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.105621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105589 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.105695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105638 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bccpw\" (UniqueName: \"kubernetes.io/projected/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-kube-api-access-bccpw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.105758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105693 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.105758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.105713 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.152934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.152894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:44.207076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207042 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.207370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207096 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.207766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207730 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.207875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207797 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bccpw\" (UniqueName: \"kubernetes.io/projected/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-kube-api-access-bccpw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.207875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207868 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.207996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.207896 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.208351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.208324 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.209132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.209111 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.209226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.209166 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.210625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.210596 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.210706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.210651 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.219916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.219895 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bccpw\" (UniqueName: \"kubernetes.io/projected/377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4-kube-api-access-bccpw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ztdt5\" (UID: \"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.388181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.388101 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:44.660022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.659995 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5"] Oct 31 03:04:44.662429 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:04:44.662389 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377b06b5_3c1e_4455_a1ec_e3cf9b09f8d4.slice/crio-8eceac766cb92e82bd6eda398396a9f3d395e76f35b8c3ce88c098b94702f66d WatchSource:0}: Error finding container 8eceac766cb92e82bd6eda398396a9f3d395e76f35b8c3ce88c098b94702f66d: Status 404 returned error can't find the container with id 8eceac766cb92e82bd6eda398396a9f3d395e76f35b8c3ce88c098b94702f66d Oct 31 03:04:44.798481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.798393 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:04:44.808188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:44.808155 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:45.359318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.359224 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" event={"ID":"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4","Type":"ContainerStarted","Data":"7d5b1488044d6cd4e45d5724241ce544f1d7578312bc2b4e10fa62d29b7d4b24"} Oct 31 03:04:45.359318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.359286 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" event={"ID":"377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4","Type":"ContainerStarted","Data":"8eceac766cb92e82bd6eda398396a9f3d395e76f35b8c3ce88c098b94702f66d"} Oct 31 03:04:45.378169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.378093 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" podStartSLOduration=1.11345559 podStartE2EDuration="1.378074226s" totalImagesPullingTime="264.618636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:45.377004644 +0000 UTC m=+563.232021786" watchObservedRunningTime="2026-10-31 03:04:45.378074226 +0000 UTC m=+563.233091368" Oct 31 03:04:45.388725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.388695 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:45.404764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.404731 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd"] Oct 31 03:04:45.408619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.408600 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.412182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.412155 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 03:04:45.419472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.419445 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd"] Oct 31 03:04:45.419898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.419875 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.420002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.419939 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.420060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.420041 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.420119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.420103 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.420172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.420154 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.420228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.420182 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcg87\" (UniqueName: \"kubernetes.io/projected/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-kube-api-access-gcg87\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.485985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.485942 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:45.521216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521184 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521240 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521266 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521284 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcg87\" (UniqueName: \"kubernetes.io/projected/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-kube-api-access-gcg87\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521307 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521338 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521755 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521827 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.521881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.521847 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.523565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.523545 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.523729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.523713 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.532983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.532957 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcg87\" (UniqueName: \"kubernetes.io/projected/3ef6c22d-6630-4ae2-8065-4dce4bf222c5-kube-api-access-gcg87\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-79tcd\" (UID: \"3ef6c22d-6630-4ae2-8065-4dce4bf222c5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.725828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.725792 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:45.934803 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:04:45.934707 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:04:45.998572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:45.998547 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd"] Oct 31 03:04:46.001019 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:04:46.000986 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef6c22d_6630_4ae2_8065_4dce4bf222c5.slice/crio-f39977ec9dde6b01b1e4bd1f62022031d2e6817ac11fd5fb8a6d8f8c0302c9c7 WatchSource:0}: Error finding container f39977ec9dde6b01b1e4bd1f62022031d2e6817ac11fd5fb8a6d8f8c0302c9c7: Status 404 returned error can't find the container with id f39977ec9dde6b01b1e4bd1f62022031d2e6817ac11fd5fb8a6d8f8c0302c9c7 Oct 31 03:04:46.366128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.366089 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" event={"ID":"3ef6c22d-6630-4ae2-8065-4dce4bf222c5","Type":"ContainerStarted","Data":"91b51c19b1e37e2b5f878082310a3ac69e7d996454ef5e070190e57cca9a4cb3"} Oct 31 03:04:46.366582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.366176 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" event={"ID":"3ef6c22d-6630-4ae2-8065-4dce4bf222c5","Type":"ContainerStarted","Data":"f39977ec9dde6b01b1e4bd1f62022031d2e6817ac11fd5fb8a6d8f8c0302c9c7"} Oct 31 03:04:46.389080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.389013 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" podStartSLOduration=1.1128107329999999 podStartE2EDuration="1.388992524s" totalImagesPullingTime="276.181791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:46.387396439 +0000 UTC m=+564.242413581" watchObservedRunningTime="2026-10-31 03:04:46.388992524 +0000 UTC m=+564.244009666" Oct 31 03:04:46.726202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.726167 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:46.824172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.824136 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4"] Oct 31 03:04:46.828197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.828168 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.834383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.834356 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 03:04:46.839000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.838977 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4"] Oct 31 03:04:46.904386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.904352 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:46.939774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.939741 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.939774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.939777 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.939983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.939892 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.939983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.939949 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.939983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.939975 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7k7\" (UniqueName: \"kubernetes.io/projected/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-kube-api-access-pf7k7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:46.940125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:46.940055 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.040884 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.040937 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7k7\" (UniqueName: \"kubernetes.io/projected/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-kube-api-access-pf7k7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.040989 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.041077 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.041106 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.041207 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.041914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.041735 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.045434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.041993 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.045434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.042603 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.051432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.046359 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.051432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.048448 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.060432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.060385 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7k7\" (UniqueName: \"kubernetes.io/projected/e930e90e-f96a-4890-a4f3-0e3ebaf83a66-kube-api-access-pf7k7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4\" (UID: \"e930e90e-f96a-4890-a4f3-0e3ebaf83a66\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.143292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.143255 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:47.419929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.419893 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4"] Oct 31 03:04:47.606085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:47.606047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:48.375598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:48.375558 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" event={"ID":"e930e90e-f96a-4890-a4f3-0e3ebaf83a66","Type":"ContainerStarted","Data":"d90539ee87dd8c6c4f224393c67e49e27489d4af2cbd5c161dcce271faa5b58f"} Oct 31 03:04:48.375780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:48.375615 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" event={"ID":"e930e90e-f96a-4890-a4f3-0e3ebaf83a66","Type":"ContainerStarted","Data":"5f57f5b0f17bcb1a8361a84c8f91cb8764dcfff3d1ed5108def7dfdd4e6182b6"} Oct 31 03:04:48.397011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:48.396963 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" podStartSLOduration=2.13803235 podStartE2EDuration="2.396950535s" totalImagesPullingTime="258.918185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:04:48.395510522 +0000 UTC m=+566.250527674" watchObservedRunningTime="2026-10-31 03:04:48.396950535 +0000 UTC m=+566.251967670" Oct 31 03:04:48.534141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:48.534105 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:04:48.943616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:48.943577 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:04:49.148482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:49.147493 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:49.269982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:49.269894 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:49.283143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:49.283110 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:49.396946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:49.396916 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:04:53.458087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:53.458049 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:04:54.389139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:54.389103 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:54.401665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:54.401636 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:54.418703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:54.418674 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" Oct 31 03:04:55.726058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:55.726024 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:55.738728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:55.738704 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:56.426692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:56.426660 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-79tcd" Oct 31 03:04:57.144318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:57.144284 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:57.156448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:57.156406 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:04:57.431435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:04:57.431377 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4" Oct 31 03:05:01.655904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.655872 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2"] Oct 31 03:05:01.662584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.662564 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.666041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.666019 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zfwtw\"" Oct 31 03:05:01.675558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.675533 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd"] Oct 31 03:05:01.680098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.680075 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2"] Oct 31 03:05:01.680224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.680167 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.695330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.695287 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd"] Oct 31 03:05:01.781577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781541 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.781777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781581 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lg7\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-kube-api-access-g8lg7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.781777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781660 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.781777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781703 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.781934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781810 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.781934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781838 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.781934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781907 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781935 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.782061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.781971 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782044 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4q47\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-kube-api-access-k4q47\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782080 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782097 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.782167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782145 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50da3361-2849-446a-85a7-f0d8c96ef051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.782167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782162 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50da3361-2849-446a-85a7-f0d8c96ef051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.782339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782220 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782245 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.782339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782325 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.782481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.782343 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883389 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50da3361-2849-446a-85a7-f0d8c96ef051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883457 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50da3361-2849-446a-85a7-f0d8c96ef051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883485 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.883727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883626 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.883727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883675 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883691 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883738 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883758 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lg7\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-kube-api-access-g8lg7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.883942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883793 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.883942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883873 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.883942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883891 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883972 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.883999 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884039 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884040 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884063 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884104 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884135 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4q47\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-kube-api-access-k4q47\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884174 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884199 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884219 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884288 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50da3361-2849-446a-85a7-f0d8c96ef051-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884562 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884597 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884601 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.884940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884732 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884842 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.884940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.884914 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.886308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.886274 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.886410 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.886285 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50da3361-2849-446a-85a7-f0d8c96ef051-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.886821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.886798 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.886900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.886829 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50da3361-2849-446a-85a7-f0d8c96ef051-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.896005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.895978 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.896210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.896189 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.896278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.896262 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lg7\" (UniqueName: \"kubernetes.io/projected/50da3361-2849-446a-85a7-f0d8c96ef051-kube-api-access-g8lg7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-qbtsd\" (UID: \"50da3361-2849-446a-85a7-f0d8c96ef051\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:01.896577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.896560 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4q47\" (UniqueName: \"kubernetes.io/projected/b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb-kube-api-access-k4q47\") pod \"maas-default-gateway-openshift-default-6db586dd6c-gc7b2\" (UID: \"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.977847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.977761 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:01.994104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:01.994074 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:02.259209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.259124 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2"] Oct 31 03:05:02.261916 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:05:02.261881 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29c7ebc_a98a_484f_b69f_dc57eaa6ddfb.slice/crio-9cbb1780c1d7ecc9432bf47d2de5ba2f73178901afb565a8abac9d625113d9f0 WatchSource:0}: Error finding container 9cbb1780c1d7ecc9432bf47d2de5ba2f73178901afb565a8abac9d625113d9f0: Status 404 returned error can't find the container with id 9cbb1780c1d7ecc9432bf47d2de5ba2f73178901afb565a8abac9d625113d9f0 Oct 31 03:05:02.264079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.264049 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.264155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.264110 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.264155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.264138 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.279744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.279717 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd"] Oct 31 03:05:02.281477 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:05:02.281445 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50da3361_2849_446a_85a7_f0d8c96ef051.slice/crio-6ca6a5f431164d2d0a0f24cb8ac20539ceef6eb29bd768a5141d87a6fa5d5a4e WatchSource:0}: Error finding container 6ca6a5f431164d2d0a0f24cb8ac20539ceef6eb29bd768a5141d87a6fa5d5a4e: Status 404 returned error can't find the container with id 6ca6a5f431164d2d0a0f24cb8ac20539ceef6eb29bd768a5141d87a6fa5d5a4e Oct 31 03:05:02.283699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.283668 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.283783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.283733 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.283783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.283772 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:02.449887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.449849 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" event={"ID":"50da3361-2849-446a-85a7-f0d8c96ef051","Type":"ContainerStarted","Data":"9fb649f360aaa1b744a5711bb62b089e39d67c811d278446b38f1085c1b51cfc"} Oct 31 03:05:02.450047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.449932 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" event={"ID":"50da3361-2849-446a-85a7-f0d8c96ef051","Type":"ContainerStarted","Data":"6ca6a5f431164d2d0a0f24cb8ac20539ceef6eb29bd768a5141d87a6fa5d5a4e"} Oct 31 03:05:02.452163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.452136 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" event={"ID":"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb","Type":"ContainerStarted","Data":"29e2b8f9b5d93243d309d55cc0c6665ca4595f84cfd46f8d9895f1c768ab3461"} Oct 31 03:05:02.452312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.452184 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" event={"ID":"b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb","Type":"ContainerStarted","Data":"9cbb1780c1d7ecc9432bf47d2de5ba2f73178901afb565a8abac9d625113d9f0"} Oct 31 03:05:02.474696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.474644 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podStartSLOduration=1.474627184 podStartE2EDuration="1.474627184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:02.472574195 +0000 UTC m=+580.327591342" watchObservedRunningTime="2026-10-31 03:05:02.474627184 +0000 UTC m=+580.329644326" Oct 31 03:05:02.497172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.497126 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podStartSLOduration=1.497108779 podStartE2EDuration="1.497108779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:02.494880419 +0000 UTC m=+580.349897585" watchObservedRunningTime="2026-10-31 03:05:02.497108779 +0000 UTC m=+580.352125921" Oct 31 03:05:02.497339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.497182 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:02.979558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.979511 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:03.002439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:02.995526 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:03.518746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.518695 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-848575bfd6-5xvtk"] Oct 31 03:05:03.522994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.522968 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.529366 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.529339 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-848575bfd6-5xvtk"] Oct 31 03:05:03.622905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.622822 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fd99a4a-9462-4af7-9469-60814b1efc22-tls-cert\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.623500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.623463 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92lvj\" (UniqueName: \"kubernetes.io/projected/8fd99a4a-9462-4af7-9469-60814b1efc22-kube-api-access-92lvj\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.725024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.724986 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fd99a4a-9462-4af7-9469-60814b1efc22-tls-cert\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.725220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.725091 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92lvj\" (UniqueName: \"kubernetes.io/projected/8fd99a4a-9462-4af7-9469-60814b1efc22-kube-api-access-92lvj\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.727732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.727706 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fd99a4a-9462-4af7-9469-60814b1efc22-tls-cert\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.736917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.736892 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92lvj\" (UniqueName: \"kubernetes.io/projected/8fd99a4a-9462-4af7-9469-60814b1efc22-kube-api-access-92lvj\") pod \"authorino-848575bfd6-5xvtk\" (UID: \"8fd99a4a-9462-4af7-9469-60814b1efc22\") " pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.835299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.835210 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848575bfd6-5xvtk" Oct 31 03:05:03.996435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.988536 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:03.996435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:03.988602 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:04.019285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.018852 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:04.019285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.018934 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:04.150677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.150647 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-848575bfd6-5xvtk"] Oct 31 03:05:04.153624 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:05:04.153595 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fd99a4a_9462_4af7_9469_60814b1efc22.slice/crio-c33021e1e7f16fb532e82454c9cfbbea454f5d468fab18a2faf51bf5c8dac432 WatchSource:0}: Error finding container c33021e1e7f16fb532e82454c9cfbbea454f5d468fab18a2faf51bf5c8dac432: Status 404 returned error can't find the container with id c33021e1e7f16fb532e82454c9cfbbea454f5d468fab18a2faf51bf5c8dac432 Oct 31 03:05:04.463614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.463529 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-848575bfd6-5xvtk" event={"ID":"8fd99a4a-9462-4af7-9469-60814b1efc22","Type":"ContainerStarted","Data":"c33021e1e7f16fb532e82454c9cfbbea454f5d468fab18a2faf51bf5c8dac432"} Oct 31 03:05:04.980262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.979814 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:04.980262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.979876 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:04.995931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.995815 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:04.995931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:04.995876 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:05.989443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:05.987842 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:05.989443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:05.987908 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:06.011439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.005400 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:06.011439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.005490 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:06.524735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.524622 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-848575bfd6-5xvtk" event={"ID":"8fd99a4a-9462-4af7-9469-60814b1efc22","Type":"ContainerStarted","Data":"5a9f7e393be73d8fcc09f80ed1c82ea31c9f3c8967460431046d8867c672645f"} Oct 31 03:05:06.542671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.541654 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-848575bfd6-5xvtk" podStartSLOduration=2.290435536 podStartE2EDuration="3.541640506s" totalImagesPullingTime="1.25120497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:06.540468249 +0000 UTC m=+584.395485385" watchObservedRunningTime="2026-10-31 03:05:06.541640506 +0000 UTC m=+584.396657646" Oct 31 03:05:06.611499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.598550 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:05:06.611499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.598832 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7585446f65-hdv7j" podUID="99e777cc-346f-4b0d-9f3a-beae5a07df65" containerName="authorino" containerID="cri-o://bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd" gracePeriod=30 Oct 31 03:05:07.001442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.978824 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:07.001442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.978890 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:07.001442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.997802 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:07.001442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:06.997859 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:07.020697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.020400 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:05:07.062868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.062830 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\" (UniqueName: \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\") pod \"99e777cc-346f-4b0d-9f3a-beae5a07df65\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " Oct 31 03:05:07.063055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.062885 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\" (UniqueName: \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\") pod \"99e777cc-346f-4b0d-9f3a-beae5a07df65\" (UID: \"99e777cc-346f-4b0d-9f3a-beae5a07df65\") " Oct 31 03:05:07.066224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.066186 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw" pod "99e777cc-346f-4b0d-9f3a-beae5a07df65" (UID: "99e777cc-346f-4b0d-9f3a-beae5a07df65"). InnerVolumeSpecName "kube-api-access-rl2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:05:07.131084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.131048 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert" pod "99e777cc-346f-4b0d-9f3a-beae5a07df65" (UID: "99e777cc-346f-4b0d-9f3a-beae5a07df65"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:05:07.164526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.164488 4556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99e777cc-346f-4b0d-9f3a-beae5a07df65-tls-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:05:07.164526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.164518 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2qw\" (UniqueName: \"kubernetes.io/projected/99e777cc-346f-4b0d-9f3a-beae5a07df65-kube-api-access-rl2qw\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:05:07.333395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.333360 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:07.479439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.475986 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:07.533442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.531566 4556 generic.go:356] "Generic (PLEG): container finished" podID="99e777cc-346f-4b0d-9f3a-beae5a07df65" containerID="bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd" exitCode=0 Oct 31 03:05:07.533442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.531610 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7585446f65-hdv7j" Oct 31 03:05:07.533442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.531686 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7585446f65-hdv7j" event={"ID":"99e777cc-346f-4b0d-9f3a-beae5a07df65","Type":"ContainerDied","Data":"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd"} Oct 31 03:05:07.533442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.531735 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7585446f65-hdv7j" event={"ID":"99e777cc-346f-4b0d-9f3a-beae5a07df65","Type":"ContainerDied","Data":"51505d0b77d4c45958dcbc26f0d6427042f9aca87d5ba6af9ad975a5aed15ec3"} Oct 31 03:05:07.533442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.531764 4556 scope.go:122] "RemoveContainer" containerID="bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd" Oct 31 03:05:07.542404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.542340 4556 scope.go:122] "RemoveContainer" containerID="bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd" Oct 31 03:05:07.542920 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:05:07.542888 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd\": container with ID starting with bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd not found: ID does not exist" containerID="bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd" Oct 31 03:05:07.543022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.542918 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd"} err="failed to get container status \"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd\": rpc error: code = NotFound desc = could not find container \"bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd\": container with ID starting with bfcd6d18fcdd1bebc240fea3e17c7aafaf5bbaf192ca78d12d830df7bd8527bd not found: ID does not exist" Oct 31 03:05:07.571439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.564333 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:05:07.571439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.564396 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7585446f65-hdv7j"] Oct 31 03:05:07.643517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.643408 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:07.766631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.766557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:07.982788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.982713 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:07.983097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:07.983028 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:08.008465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.008318 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:08.008465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.008378 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:08.256003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.255967 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e777cc-346f-4b0d-9f3a-beae5a07df65" path="/var/lib/kubelet/pods/99e777cc-346f-4b0d-9f3a-beae5a07df65/volumes" Oct 31 03:05:08.441135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.441094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:08.978805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.978761 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-gc7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:05:08.979015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.978838 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" podUID="b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:05:08.994995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.994958 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-qbtsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:05:08.995131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:08.995021 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" podUID="50da3361-2849-446a-85a7-f0d8c96ef051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:05:10.808438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.805922 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:10.808438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.807431 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:10.809649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.809621 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-qbtsd" Oct 31 03:05:10.898588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.898549 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp"] Oct 31 03:05:10.899521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.899497 4556 cpu_manager.go:414] "containerMap: removing container" podUID="99e777cc-346f-4b0d-9f3a-beae5a07df65" containerName="authorino" Oct 31 03:05:10.899892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.899871 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e777cc-346f-4b0d-9f3a-beae5a07df65" containerName="authorino" Oct 31 03:05:10.900272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.900252 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e777cc-346f-4b0d-9f3a-beae5a07df65" containerName="authorino" Oct 31 03:05:10.903198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.899752 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:10.905266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.905245 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:10.913852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.913811 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp"] Oct 31 03:05:10.916331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.916301 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6nw55\"" Oct 31 03:05:10.918840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.918684 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 03:05:10.938473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.938025 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:10.938473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.938460 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:10.939568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:10.939547 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-gc7b2" Oct 31 03:05:11.047947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.047889 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.048300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.048285 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjs9w\" (UniqueName: \"kubernetes.io/projected/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-kube-api-access-vjs9w\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.149947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.149909 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.150376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.150353 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjs9w\" (UniqueName: \"kubernetes.io/projected/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-kube-api-access-vjs9w\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.153673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.151751 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.162786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.162744 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjs9w\" (UniqueName: \"kubernetes.io/projected/b3d3cba7-e7fe-43ae-89a2-3fce419f1232-kube-api-access-vjs9w\") pod \"payload-pre-processing-7b98bb678c-6jkdp\" (UID: \"b3d3cba7-e7fe-43ae-89a2-3fce419f1232\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.229785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.229743 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:11.522922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.522890 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp"] Oct 31 03:05:11.526660 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:05:11.526628 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d3cba7_e7fe_43ae_89a2_3fce419f1232.slice/crio-cc2cb88e3dd4d60652987bc3508891cfac21bbb4fc7a401492d1f9943c809854 WatchSource:0}: Error finding container cc2cb88e3dd4d60652987bc3508891cfac21bbb4fc7a401492d1f9943c809854: Status 404 returned error can't find the container with id cc2cb88e3dd4d60652987bc3508891cfac21bbb4fc7a401492d1f9943c809854 Oct 31 03:05:11.555935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:11.555886 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" event={"ID":"b3d3cba7-e7fe-43ae-89a2-3fce419f1232","Type":"ContainerStarted","Data":"cc2cb88e3dd4d60652987bc3508891cfac21bbb4fc7a401492d1f9943c809854"} Oct 31 03:05:12.916101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:12.915995 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:15.577569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:15.577525 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" event={"ID":"b3d3cba7-e7fe-43ae-89a2-3fce419f1232","Type":"ContainerStarted","Data":"a691cd4702d73686bc3faedb10ddcdb33b0127913c8d201a87cbb1c984871240"} Oct 31 03:05:15.578005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:15.577683 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:15.595688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:15.595620 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" podStartSLOduration=2.293239666 podStartE2EDuration="5.595601822s" totalImagesPullingTime="3.302362156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:15.59416697 +0000 UTC m=+593.449184112" watchObservedRunningTime="2026-10-31 03:05:15.595601822 +0000 UTC m=+593.450618966" Oct 31 03:05:15.935942 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:05:15.935900 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:05:16.704539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.704493 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn"] Oct 31 03:05:16.708364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.708339 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.725666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.725628 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn"] Oct 31 03:05:16.840865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.840817 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.840941 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.840983 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841055 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841073 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2sr\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-kube-api-access-6c2sr\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841096 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841115 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.841401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.841345 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.942953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.942914 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.942953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.942957 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.942975 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2sr\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-kube-api-access-6c2sr\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.942997 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943017 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943080 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943122 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943168 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943236 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943404 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943492 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943563 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943594 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.943961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.943940 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.945459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.945437 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.945724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.945703 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.954953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.954891 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:16.955093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:16.954976 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2sr\" (UniqueName: \"kubernetes.io/projected/3fa8ea03-5ae9-4b78-948e-96ee4132a113-kube-api-access-6c2sr\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5bjzn\" (UID: \"3fa8ea03-5ae9-4b78-948e-96ee4132a113\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:17.022179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.022138 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:17.305103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.305075 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn"] Oct 31 03:05:17.307897 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:05:17.307862 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa8ea03_5ae9_4b78_948e_96ee4132a113.slice/crio-591908246aa0e758d4011cd9ed646b22ba5dd4f160edd58920986b0a2dcb74f9 WatchSource:0}: Error finding container 591908246aa0e758d4011cd9ed646b22ba5dd4f160edd58920986b0a2dcb74f9: Status 404 returned error can't find the container with id 591908246aa0e758d4011cd9ed646b22ba5dd4f160edd58920986b0a2dcb74f9 Oct 31 03:05:17.310218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.310187 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:17.310300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.310253 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:17.310300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.310279 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:05:17.592627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.592541 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" event={"ID":"3fa8ea03-5ae9-4b78-948e-96ee4132a113","Type":"ContainerStarted","Data":"b58280309c892817ca76a1e4512114c7a014e46298ca08261170936db771deea"} Oct 31 03:05:17.592627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.592594 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" event={"ID":"3fa8ea03-5ae9-4b78-948e-96ee4132a113","Type":"ContainerStarted","Data":"591908246aa0e758d4011cd9ed646b22ba5dd4f160edd58920986b0a2dcb74f9"} Oct 31 03:05:17.615214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:17.615149 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podStartSLOduration=1.615133645 podStartE2EDuration="1.615133645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:05:17.614189895 +0000 UTC m=+595.469207037" watchObservedRunningTime="2026-10-31 03:05:17.615133645 +0000 UTC m=+595.470150785" Oct 31 03:05:18.023029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:18.022991 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:18.024558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:18.024531 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:18.024671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:18.024579 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:19.022875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:19.022836 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:19.023053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:19.022896 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:20.023227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:20.023185 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:20.023699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:20.023244 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:21.023349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:21.023310 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:21.023801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:21.023372 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:22.023468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:22.023391 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:22.023895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:22.023489 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:23.023390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:23.023353 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:23.023594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:23.023449 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:24.022840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:24.022793 4556 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5bjzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 03:05:24.023055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:24.022857 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" podUID="3fa8ea03-5ae9-4b78-948e-96ee4132a113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 03:05:25.631472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:25.631409 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:25.631938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:25.631831 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:25.632697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:25.632678 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5bjzn" Oct 31 03:05:26.587048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:26.587018 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jkdp" Oct 31 03:05:37.953326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:37.953292 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:38.467937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:38.467897 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:40.298164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:40.298129 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:40.444058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:40.444022 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:41.971458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:41.971406 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:42.096718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:05:42.096681 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:05:45.934591 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:05:45.934550 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:06:07.507657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:07.507614 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:08.486185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:08.486153 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.039140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.039097 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.177669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.177637 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.430615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.430580 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:06:12.434280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.434257 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.441368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.441344 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebea4-openshift-default-dockercfg-wg8mv\"" Oct 31 03:06:12.451461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.451437 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:06:12.493377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493345 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493391 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493505 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbtx\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493540 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493579 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493596 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493731 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493779 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.493910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.493887 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.526820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.526772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.595172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595135 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595189 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595233 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595272 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbtx\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595303 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595349 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595375 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595442 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595482 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595610 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595625 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.595891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595830 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.596055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.595994 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.596359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.596334 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.597700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.597682 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.598153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.598133 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.613151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.613115 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbtx\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.614755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.614608 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.664753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.664706 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.713825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.713711 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:06:12.719321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.719296 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.733170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.733142 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:06:12.743228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.743200 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-99bc97-openshift-default-dockercfg-pzvq9\"" Oct 31 03:06:12.749606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.749579 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:12.799857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.799813 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.799890 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.799920 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.799974 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.800027 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.800053 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh426\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.800112 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.800173 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.800232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.800197 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.829442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.829198 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:12.900741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.900707 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:06:12.901263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901159 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901224 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901249 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901307 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901351 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901378 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901437 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901484 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.901777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.901513 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh426\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.902583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.902559 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.902978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.902952 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.903111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.903085 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.903366 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.903336 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.903647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.903611 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.904714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.904687 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.905961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.905924 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.908038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.907980 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:12.914038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.914016 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh426\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.916989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.916946 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:12.917842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.917802 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-edb9e6-openshift-default-dockercfg-n4zc9\"" Oct 31 03:06:12.920179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.920151 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:06:12.971462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:12.971361 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:13.002025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.001996 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ct6\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002055 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002145 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002267 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002304 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002371 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002408 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002444 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.002598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.002570 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.039100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.039067 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:13.057074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.057049 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:06:13.058765 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:13.058695 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f16360_e558_4069_99ff_a4cfd951257f.slice/crio-74de6e23dfb65afa1f19d0b5286e03e72636f5e3e80932310ef166da931462b5 WatchSource:0}: Error finding container 74de6e23dfb65afa1f19d0b5286e03e72636f5e3e80932310ef166da931462b5: Status 404 returned error can't find the container with id 74de6e23dfb65afa1f19d0b5286e03e72636f5e3e80932310ef166da931462b5 Oct 31 03:06:13.061080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.061051 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.061145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.061116 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.061185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.061144 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.103503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103473 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103533 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103583 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103614 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103657 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103705 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ct6\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103783 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103812 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.103906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.103892 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.104295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.104262 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.104997 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.105250 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.105509 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.106153 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.106519 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.106873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.106835 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:13.107198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.107182 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.117085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.117064 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.117233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.117213 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ct6\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.228835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.228732 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:13.233672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.233642 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:13.324220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.324194 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:06:13.326228 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:13.326192 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198e0fdd_69de_4242_83f7_bee12900dc7c.slice/crio-666aa2a7cee6b6621e666f06ce18beb1198878f8fdbcdf69bc037924854f9045 WatchSource:0}: Error finding container 666aa2a7cee6b6621e666f06ce18beb1198878f8fdbcdf69bc037924854f9045: Status 404 returned error can't find the container with id 666aa2a7cee6b6621e666f06ce18beb1198878f8fdbcdf69bc037924854f9045 Oct 31 03:06:13.329246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.329204 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.329347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.329296 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.329397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.329380 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.370400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.370372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:13.514489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.514461 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:06:13.516060 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:13.516034 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071bbc6_f4be_4a6c_a499_8eb27cc71926.slice/crio-33502efd8427eb600998465eef0b21fa87f5fe377cbaf8654f8654ed822d0ced WatchSource:0}: Error finding container 33502efd8427eb600998465eef0b21fa87f5fe377cbaf8654f8654ed822d0ced: Status 404 returned error can't find the container with id 33502efd8427eb600998465eef0b21fa87f5fe377cbaf8654f8654ed822d0ced Oct 31 03:06:13.518782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.518754 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.518866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.518821 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.518866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.518848 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:13.840591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.840507 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerStarted","Data":"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347"} Oct 31 03:06:13.840591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.840568 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerStarted","Data":"666aa2a7cee6b6621e666f06ce18beb1198878f8fdbcdf69bc037924854f9045"} Oct 31 03:06:13.842162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.842128 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerStarted","Data":"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919"} Oct 31 03:06:13.842329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.842184 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerStarted","Data":"33502efd8427eb600998465eef0b21fa87f5fe377cbaf8654f8654ed822d0ced"} Oct 31 03:06:13.843711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.843684 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerStarted","Data":"1080c06b96636257f917c67329925c33db7746d873b7f51e3f66625120c44817"} Oct 31 03:06:13.843806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.843724 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerStarted","Data":"74de6e23dfb65afa1f19d0b5286e03e72636f5e3e80932310ef166da931462b5"} Oct 31 03:06:13.885981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.885927 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podStartSLOduration=1.885911881 podStartE2EDuration="1.885911881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:13.882577888 +0000 UTC m=+651.737595040" watchObservedRunningTime="2026-10-31 03:06:13.885911881 +0000 UTC m=+651.740929067" Oct 31 03:06:13.915640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.915575 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podStartSLOduration=1.9155581449999999 podStartE2EDuration="1.915558145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:13.913800813 +0000 UTC m=+651.768818164" watchObservedRunningTime="2026-10-31 03:06:13.915558145 +0000 UTC m=+651.770575288" Oct 31 03:06:13.978538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:13.978471 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podStartSLOduration=1.9783250159999999 podStartE2EDuration="1.978325016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:13.977973765 +0000 UTC m=+651.832990918" watchObservedRunningTime="2026-10-31 03:06:13.978325016 +0000 UTC m=+651.833342158" Oct 31 03:06:14.039289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.039250 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:14.044750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.044728 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:14.229215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.229181 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:14.234607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.234582 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:14.750236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.750204 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:14.755733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.755710 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:14.848439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.848379 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:14.848608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.848465 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:14.848832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.848779 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:14.849392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.849372 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:14.849517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.849481 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:14.849574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:14.849562 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:15.807522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:15.807482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:15.934710 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:06:15.934671 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:06:15.947632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:15.947596 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:16.376031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:16.375982 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:16.520500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:16.520456 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:16.652293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:16.652265 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:16.905022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:16.904953 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.053332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.053297 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.205706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.205623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.478832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.478753 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.614531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.614495 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:06:17.620012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.619974 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.641409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.641385 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-56afbd\"" Oct 31 03:06:17.650926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.650904 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-56afbd-dockercfg-x4qvx\"" Oct 31 03:06:17.669134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.669108 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:06:17.728388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.728333 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.762212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.762130 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.762212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.762175 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjlg\" (UniqueName: \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.863245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.863200 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.863464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.863276 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjlg\" (UniqueName: \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.864034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.864010 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.868492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.868464 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:17.876012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.875985 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjlg\" (UniqueName: \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\") pod \"payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:17.933780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:17.933754 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:18.005453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.003061 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:18.214750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.214576 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:06:18.216873 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:18.216842 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76188f96_56cb_4c55_b324_205cacde847c.slice/crio-8b52cef9dce97ac115d2a1b164eee49541fc610929d92a23606b6f43a7a88f19 WatchSource:0}: Error finding container 8b52cef9dce97ac115d2a1b164eee49541fc610929d92a23606b6f43a7a88f19: Status 404 returned error can't find the container with id 8b52cef9dce97ac115d2a1b164eee49541fc610929d92a23606b6f43a7a88f19 Oct 31 03:06:18.419903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.419856 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:18.463780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.463747 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:06:18.467258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.467212 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.472552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.472526 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-bebea4\"" Oct 31 03:06:18.473179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.473156 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-bebea4-dockercfg-5sktm\"" Oct 31 03:06:18.478407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.478384 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:06:18.516001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.515973 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:06:18.519379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.519362 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.531872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.531849 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:06:18.586853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.571437 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfwz\" (UniqueName: \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.586853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.571552 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.586853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.580458 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:18.688054 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.688012 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfwz\" (UniqueName: \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.688207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.688095 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h6n\" (UniqueName: \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.688207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.688120 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.688313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.688211 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.688863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.688845 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.699910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.699892 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfwz\" (UniqueName: \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\") pod \"payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.781331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.781256 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:18.789825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.789802 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h6n\" (UniqueName: \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.789882 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.789847 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.790499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.790478 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.802490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.802461 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h6n\" (UniqueName: \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\") pod \"payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.832147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.832112 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:18.866012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.865983 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" event={"ID":"76188f96-56cb-4c55-b324-205cacde847c","Type":"ContainerStarted","Data":"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8"} Oct 31 03:06:18.866155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.866043 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" event={"ID":"76188f96-56cb-4c55-b324-205cacde847c","Type":"ContainerStarted","Data":"8b52cef9dce97ac115d2a1b164eee49541fc610929d92a23606b6f43a7a88f19"} Oct 31 03:06:18.866803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.866779 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:18.883891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:18.883748 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" podStartSLOduration=1.88373087 podStartE2EDuration="1.88373087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:18.883166824 +0000 UTC m=+656.738183963" watchObservedRunningTime="2026-10-31 03:06:18.88373087 +0000 UTC m=+656.738748013" Oct 31 03:06:19.025332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.025292 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.059156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.059106 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:06:19.062087 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:19.062058 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26adffd_9b62_4b0d_9f8f_32d994efb4d4.slice/crio-7d73aba75f7131f88b806eede4c3714eb69fbe6598ade1a1e4d0caf3ff720502 WatchSource:0}: Error finding container 7d73aba75f7131f88b806eede4c3714eb69fbe6598ade1a1e4d0caf3ff720502: Status 404 returned error can't find the container with id 7d73aba75f7131f88b806eede4c3714eb69fbe6598ade1a1e4d0caf3ff720502 Oct 31 03:06:19.133878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.133748 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:06:19.135786 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:19.135758 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ab1005_92d1_4af6_ae20_b691ce4378a8.slice/crio-771018affd0f6d354feca560d14beafc27fc19c48c36ca03345417e68fc85565 WatchSource:0}: Error finding container 771018affd0f6d354feca560d14beafc27fc19c48c36ca03345417e68fc85565: Status 404 returned error can't find the container with id 771018affd0f6d354feca560d14beafc27fc19c48c36ca03345417e68fc85565 Oct 31 03:06:19.192666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.192633 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.301112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.301079 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:06:19.304719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.304701 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.310532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.310471 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-99bc97-dockercfg-flbqc\"" Oct 31 03:06:19.313492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.313471 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-99bc97\"" Oct 31 03:06:19.316911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.316886 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:06:19.329091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.329061 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:06:19.332742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.332721 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.336808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.336783 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.341277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.341250 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:06:19.395203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.395174 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.395365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.395301 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsvj\" (UniqueName: \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.465797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.465763 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.496962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.496928 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsvj\" (UniqueName: \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.497146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.497050 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.497146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.497089 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6fn\" (UniqueName: \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.497146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.497116 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.497814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.497788 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.512847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.512822 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsvj\" (UniqueName: \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\") pod \"payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.598733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.598700 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.598909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.598747 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6fn\" (UniqueName: \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.599497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.599479 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.611988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.611962 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6fn\" (UniqueName: \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\") pod \"payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.618602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.618584 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:19.635112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.635073 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.647461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.647432 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:19.775467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.774072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.873814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.873719 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" event={"ID":"b26adffd-9b62-4b0d-9f8f-32d994efb4d4","Type":"ContainerStarted","Data":"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f"} Oct 31 03:06:19.873814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.873775 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" event={"ID":"b26adffd-9b62-4b0d-9f8f-32d994efb4d4","Type":"ContainerStarted","Data":"7d73aba75f7131f88b806eede4c3714eb69fbe6598ade1a1e4d0caf3ff720502"} Oct 31 03:06:19.874024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.873872 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:19.878680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.878652 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" event={"ID":"d5ab1005-92d1-4af6-ae20-b691ce4378a8","Type":"ContainerStarted","Data":"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c"} Oct 31 03:06:19.878825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.878704 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" event={"ID":"d5ab1005-92d1-4af6-ae20-b691ce4378a8","Type":"ContainerStarted","Data":"771018affd0f6d354feca560d14beafc27fc19c48c36ca03345417e68fc85565"} Oct 31 03:06:19.896220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.896170 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" podStartSLOduration=1.8961582510000001 podStartE2EDuration="1.896158251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:19.893684917 +0000 UTC m=+657.748702058" watchObservedRunningTime="2026-10-31 03:06:19.896158251 +0000 UTC m=+657.751175438" Oct 31 03:06:19.902566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.902530 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:06:19.906095 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:19.906064 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314bff69_cff9_4977_a24c_d9b7d65f0a40.slice/crio-e003d5ddca362cd37d267c85fc100c627bc9f5c44f34935790861c467547d9c8 WatchSource:0}: Error finding container e003d5ddca362cd37d267c85fc100c627bc9f5c44f34935790861c467547d9c8: Status 404 returned error can't find the container with id e003d5ddca362cd37d267c85fc100c627bc9f5c44f34935790861c467547d9c8 Oct 31 03:06:19.918740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.918684 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" podStartSLOduration=1.918665431 podStartE2EDuration="1.918665431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:19.916836376 +0000 UTC m=+657.771853529" watchObservedRunningTime="2026-10-31 03:06:19.918665431 +0000 UTC m=+657.773682572" Oct 31 03:06:19.933664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.933640 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:06:19.941805 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:19.941781 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f9780b_1631_4c9f_bedd_df67d21f1d83.slice/crio-2b7c886f020f4190b837bf167c134f1b557d7e97b0711c585f33a6981abfd6e7 WatchSource:0}: Error finding container 2b7c886f020f4190b837bf167c134f1b557d7e97b0711c585f33a6981abfd6e7: Status 404 returned error can't find the container with id 2b7c886f020f4190b837bf167c134f1b557d7e97b0711c585f33a6981abfd6e7 Oct 31 03:06:19.956210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.956189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:19.956637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.956614 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:06:19.961270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.961249 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:19.968744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.968722 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-924ee5\"" Oct 31 03:06:19.971243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.971217 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:06:19.985250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.985228 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:06:19.993100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.993071 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:19.998944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:19.998911 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:06:20.001146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.001120 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-924ee5-dockercfg-ctkhk\"" Oct 31 03:06:20.093448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.092670 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:20.117098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.117041 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2lg\" (UniqueName: \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.117297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.117138 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.117297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.117178 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.117297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.117230 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.218474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.218405 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.218644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.218501 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.218644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.218554 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.218759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.218680 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2lg\" (UniqueName: \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.219268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.219247 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.219533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.219499 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.239876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.239846 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2lg\" (UniqueName: \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\") pod \"payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.240378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.240359 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\") pod \"payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.290808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.290776 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:20.297258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.297231 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:20.324954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.324271 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.479712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.479635 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:20.589604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.589575 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:06:20.592119 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:20.592091 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b869789_ca41_4036_9c44_d1fbe68481df.slice/crio-fb5f76f1f14eabae96fb99472b37672493c4f746881c48967ebd1e0fbe85c213 WatchSource:0}: Error finding container fb5f76f1f14eabae96fb99472b37672493c4f746881c48967ebd1e0fbe85c213: Status 404 returned error can't find the container with id fb5f76f1f14eabae96fb99472b37672493c4f746881c48967ebd1e0fbe85c213 Oct 31 03:06:20.613890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.613868 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:20.636154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.636007 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:06:20.638132 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:20.638104 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952f64f5_b2ba_46d9_b45a_c5d3ab037827.slice/crio-b705c6f4e8eb0cf9a239cf7faea63f405e2bfa7572569c8038cadf2a54a6dbfe WatchSource:0}: Error finding container b705c6f4e8eb0cf9a239cf7faea63f405e2bfa7572569c8038cadf2a54a6dbfe: Status 404 returned error can't find the container with id b705c6f4e8eb0cf9a239cf7faea63f405e2bfa7572569c8038cadf2a54a6dbfe Oct 31 03:06:20.881806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.881725 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:06:20.886469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.886447 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:20.886705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.886677 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" event={"ID":"952f64f5-b2ba-46d9-b45a-c5d3ab037827","Type":"ContainerStarted","Data":"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc"} Oct 31 03:06:20.886790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.886737 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:20.886790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.886761 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" event={"ID":"952f64f5-b2ba-46d9-b45a-c5d3ab037827","Type":"ContainerStarted","Data":"b705c6f4e8eb0cf9a239cf7faea63f405e2bfa7572569c8038cadf2a54a6dbfe"} Oct 31 03:06:20.888326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.888295 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" event={"ID":"97f9780b-1631-4c9f-bedd-df67d21f1d83","Type":"ContainerStarted","Data":"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5"} Oct 31 03:06:20.888466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.888342 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:20.888466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.888358 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" event={"ID":"97f9780b-1631-4c9f-bedd-df67d21f1d83","Type":"ContainerStarted","Data":"2b7c886f020f4190b837bf167c134f1b557d7e97b0711c585f33a6981abfd6e7"} Oct 31 03:06:20.889930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.889909 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" event={"ID":"314bff69-cff9-4977-a24c-d9b7d65f0a40","Type":"ContainerStarted","Data":"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72"} Oct 31 03:06:20.890042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.889950 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" event={"ID":"314bff69-cff9-4977-a24c-d9b7d65f0a40","Type":"ContainerStarted","Data":"e003d5ddca362cd37d267c85fc100c627bc9f5c44f34935790861c467547d9c8"} Oct 31 03:06:20.890042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.889997 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:20.891369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.891343 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" event={"ID":"4b869789-ca41-4036-9c44-d1fbe68481df","Type":"ContainerStarted","Data":"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e"} Oct 31 03:06:20.891488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.891388 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" event={"ID":"4b869789-ca41-4036-9c44-d1fbe68481df","Type":"ContainerStarted","Data":"fb5f76f1f14eabae96fb99472b37672493c4f746881c48967ebd1e0fbe85c213"} Oct 31 03:06:20.891573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.891556 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:20.893571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.893549 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7d8daf\"" Oct 31 03:06:20.896574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.896514 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:06:20.899236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.899215 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:20.925106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.925064 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" podStartSLOduration=1.925051767 podStartE2EDuration="1.925051767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.923184306 +0000 UTC m=+658.778201448" watchObservedRunningTime="2026-10-31 03:06:20.925051767 +0000 UTC m=+658.780068908" Oct 31 03:06:20.940658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.940604 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" podStartSLOduration=1.94058676 podStartE2EDuration="1.94058676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.939572316 +0000 UTC m=+658.794589456" watchObservedRunningTime="2026-10-31 03:06:20.94058676 +0000 UTC m=+658.795603902" Oct 31 03:06:20.958398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.958360 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" podStartSLOduration=1.958347837 podStartE2EDuration="1.958347837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.956125574 +0000 UTC m=+658.811142725" watchObservedRunningTime="2026-10-31 03:06:20.958347837 +0000 UTC m=+658.813364976" Oct 31 03:06:20.982007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:20.981954 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" podStartSLOduration=1.981939103 podStartE2EDuration="1.981939103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:20.980305333 +0000 UTC m=+658.835322474" watchObservedRunningTime="2026-10-31 03:06:20.981939103 +0000 UTC m=+658.836956244" Oct 31 03:06:21.031750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.031720 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:21.033936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.033915 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.034378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.034349 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q8n\" (UniqueName: \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.138167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.138071 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.138910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.138880 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q8n\" (UniqueName: \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.140451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.140409 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.151633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.151590 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q8n\" (UniqueName: \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\") pod \"payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.212180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.212145 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.326910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.326874 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:21.485537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.485511 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:06:21.488141 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:21.488098 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da57060_3c69_42e2_b7c7_05047706a00a.slice/crio-cee8dc394631d03ae836c822728a5721b79499cc67cf03c1d6e53608abeae6d7 WatchSource:0}: Error finding container cee8dc394631d03ae836c822728a5721b79499cc67cf03c1d6e53608abeae6d7: Status 404 returned error can't find the container with id cee8dc394631d03ae836c822728a5721b79499cc67cf03c1d6e53608abeae6d7 Oct 31 03:06:21.595807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.595770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:21.748356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.748235 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:21.781024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.780989 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:21.898508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.898453 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" event={"ID":"2da57060-3c69-42e2-b7c7-05047706a00a","Type":"ContainerStarted","Data":"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931"} Oct 31 03:06:21.898508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.898510 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" event={"ID":"2da57060-3c69-42e2-b7c7-05047706a00a","Type":"ContainerStarted","Data":"cee8dc394631d03ae836c822728a5721b79499cc67cf03c1d6e53608abeae6d7"} Oct 31 03:06:21.898750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.898641 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:21.899170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.899145 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:21.916784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:21.916706 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" podStartSLOduration=1.9166883399999999 podStartE2EDuration="1.91668834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:21.915377618 +0000 UTC m=+659.770394759" watchObservedRunningTime="2026-10-31 03:06:21.91668834 +0000 UTC m=+659.771705482" Oct 31 03:06:22.238436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:22.234101 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:22.361177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:22.361137 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:22.485919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:22.485889 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:23.123928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:23.123896 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:23.247069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:23.247030 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:24.032087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:24.032046 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:24.162364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:24.162336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:24.991570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:24.991530 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:25.134734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:25.134687 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:25.824452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:25.824403 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:25.878046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:25.877994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:26.925289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.925251 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:06:26.930524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.930501 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:26.936939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.936907 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-7d8daf-serving-cert\"" Oct 31 03:06:26.937808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.937706 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:06:26.939678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.939627 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:06:26.941847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.941825 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:06:26.942029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.942015 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:06:26.942215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:26.942202 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:06:27.005689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.005644 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrp6\" (UniqueName: \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.005937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.005903 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.006069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.006053 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.006187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.006175 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.023913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.023877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:27.107296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.107250 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.107461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.107337 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrp6\" (UniqueName: \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.107461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.107382 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.107461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.107448 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.108449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.108407 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.109961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.109938 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.110351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.110330 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.119391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.119366 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrp6\" (UniqueName: \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\") pod \"maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.154749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.154719 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:27.258111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.258012 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:27.295436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.282862 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:27.851809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.851622 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:27.961216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.961188 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:06:27.963340 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:27.963313 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1e5758_cd32_48ae_98ea_d6bc07457dc3.slice/crio-591327e37afdabe333c7c9a19c45deb5a08b6dc89dfb6522da00ac922fb943fd WatchSource:0}: Error finding container 591327e37afdabe333c7c9a19c45deb5a08b6dc89dfb6522da00ac922fb943fd: Status 404 returned error can't find the container with id 591327e37afdabe333c7c9a19c45deb5a08b6dc89dfb6522da00ac922fb943fd Oct 31 03:06:27.964756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:27.964739 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:06:28.639505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:28.639367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:28.959220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:28.959180 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" event={"ID":"7d1e5758-cd32-48ae-98ea-d6bc07457dc3","Type":"ContainerStarted","Data":"591327e37afdabe333c7c9a19c45deb5a08b6dc89dfb6522da00ac922fb943fd"} Oct 31 03:06:29.346036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:29.345847 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:29.953439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:29.951102 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:30.036501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:30.036390 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:30.918564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:30.917583 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:06:30.932068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:30.931983 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:30.953150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:30.949180 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:06:31.862096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:31.862010 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:31.985437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:31.961612 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:06:31.985437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:31.968622 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:06:32.009444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:31.998435 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:06:32.018647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:32.018613 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:06:32.637494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:32.635268 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:32.910282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:32.910174 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:06:32.912906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:32.912874 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:06:33.584922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:33.584885 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:34.420599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:34.420383 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:35.125135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:35.125103 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:35.956761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:35.956663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:36.092917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.092881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:36.427915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.427875 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" event={"ID":"7d1e5758-cd32-48ae-98ea-d6bc07457dc3","Type":"ContainerStarted","Data":"fec643931cddc172202930556e002693660fc94e7676bc49d6828bb1bf3d400f"} Oct 31 03:06:36.428359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.427938 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:36.449373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.449323 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" podStartSLOduration=2.487238612 podStartE2EDuration="10.449309018s" totalImagesPullingTime="7.962070406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:36.447196168 +0000 UTC m=+674.302213336" watchObservedRunningTime="2026-10-31 03:06:36.449309018 +0000 UTC m=+674.304326184" Oct 31 03:06:36.609091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.609047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:36.742286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.742197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:36.869457 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.029109 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.179133 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.707606 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.777805 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.816031 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.816122 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.820319 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"kube-root-ca.crt\"" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.820360 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"openshift-service-ca.crt\"" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.820392 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"default-dockercfg-gdkj4\"" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.820557 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-56afbd\"/\"simulator-w5-56afbd-kserve-self-signed-certs\"" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.841560 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.952920 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.952966 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8vd\" (UniqueName: \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.952984 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.953108 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.953163 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.953207 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:37.999370 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.054357 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.055376 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.056272 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.056527 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8vd\" (UniqueName: \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.056938 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.057158 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.059804 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.061170 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.061301 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.064924 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.069559 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8vd\" (UniqueName: \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.072712 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\") pod \"simulator-w5-56afbd-kserve-f88776d94-nh6mz\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.149368 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.291094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:38.770552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:38.449704 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:06:39.226812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.226748 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:39.445062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.445023 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" event={"ID":"51cef924-2747-43a9-a93a-f40d70e3cdaf","Type":"ContainerStarted","Data":"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810"} Oct 31 03:06:39.445258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.445072 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" event={"ID":"51cef924-2747-43a9-a93a-f40d70e3cdaf","Type":"ContainerStarted","Data":"4092732b6276e192a086ec2ec33e9fdac6a210d4f03fabfebcd60ef88066aefb"} Oct 31 03:06:39.446590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.446565 4556 generic.go:356] "Generic (PLEG): container finished" podID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerID="1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347" exitCode=0 Oct 31 03:06:39.446720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.446642 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerDied","Data":"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347"} Oct 31 03:06:39.446986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.446970 4556 scope.go:122] "RemoveContainer" containerID="1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347" Oct 31 03:06:39.448260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448237 4556 generic.go:356] "Generic (PLEG): container finished" podID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerID="c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919" exitCode=0 Oct 31 03:06:39.448349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448313 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerDied","Data":"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919"} Oct 31 03:06:39.448430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448344 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.448497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448438 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.448497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448482 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.448706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.448683 4556 scope.go:122] "RemoveContainer" containerID="c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919" Oct 31 03:06:39.449962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.449939 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.450045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.450010 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.450108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.450046 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.450108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.450085 4556 generic.go:356] "Generic (PLEG): container finished" podID="27f16360-e558-4069-99ff-a4cfd951257f" containerID="1080c06b96636257f917c67329925c33db7746d873b7f51e3f66625120c44817" exitCode=0 Oct 31 03:06:39.450219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.450106 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerDied","Data":"1080c06b96636257f917c67329925c33db7746d873b7f51e3f66625120c44817"} Oct 31 03:06:39.450582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.450501 4556 scope.go:122] "RemoveContainer" containerID="1080c06b96636257f917c67329925c33db7746d873b7f51e3f66625120c44817" Oct 31 03:06:39.451633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.451602 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.451728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.451670 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.451728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.451715 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:06:39.462724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.462685 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podStartSLOduration=2.462645385 podStartE2EDuration="2.462645385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:39.462172377 +0000 UTC m=+677.317189521" watchObservedRunningTime="2026-10-31 03:06:39.462645385 +0000 UTC m=+677.317662526" Oct 31 03:06:39.639358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:39.639324 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:40.151557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.151517 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:40.260708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.260615 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:40.464376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.464280 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerStarted","Data":"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58"} Oct 31 03:06:40.466730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.466694 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerStarted","Data":"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b"} Oct 31 03:06:40.473147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.473118 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerStarted","Data":"98a8959ce7a195b422e71ffdb06ad395ead49b714dd6a11cae6cbb850605c189"} Oct 31 03:06:40.750931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.750792 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:40.751897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.751862 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:40.752056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:40.751921 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:41.039708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.039585 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:41.043543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.043507 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 03:06:41.043699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.043565 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 03:06:41.239502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.235294 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:41.240225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.235536 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:41.240434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.235606 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:06:41.240602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.240577 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:06:41.717443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.716877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:41.750962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.750920 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:41.751109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.750973 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:41.845554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:41.845515 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.042504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.035434 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.042504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.039860 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 03:06:42.042504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.039917 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 03:06:42.163624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.163580 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.231447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.230853 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:06:42.231447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.230915 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:06:42.367087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.366863 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.461242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.461208 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:06:42.564904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.564873 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.747374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.747334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:42.773438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.751395 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:42.773748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.773712 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:42.906870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:42.905697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:43.060493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.055829 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 03:06:43.060493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.055892 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 03:06:43.101438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.100557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:43.236437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.229827 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:06:43.236437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.229892 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:06:43.751307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.750844 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:43.751307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:43.750909 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:44.060790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.060647 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:44.087424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.067716 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 03:06:44.087424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.067783 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 03:06:44.261776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.261741 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:06:44.262041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.262011 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:06:44.353497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.269186 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:06:44.362086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.362048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:44.517635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.517596 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:06:44.517904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.517768 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.547878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.547847 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"default-dockercfg-cwqcr\"" Oct 31 03:06:44.556440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.548782 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"kube-root-ca.crt\"" Oct 31 03:06:44.560835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.560646 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"simulator-w2-85cb59-kserve-self-signed-certs\"" Oct 31 03:06:44.560835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.560704 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-85cb59\"/\"openshift-service-ca.crt\"" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683575 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42d2\" (UniqueName: \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683660 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683702 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683753 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683793 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.684191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.683909 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.751648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.751600 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:44.752302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.752266 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:44.785182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.785145 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.785595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.785559 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.785881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.785847 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42d2\" (UniqueName: \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.786028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.786011 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.786171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.786153 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.789434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.786316 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.790184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.790162 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.802214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.802183 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.813545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.802806 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.834106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.830234 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.850435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.837010 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.850435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.841070 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42d2\" (UniqueName: \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\") pod \"simulator-w2-85cb59-kserve-74f449fdb9-sncmf\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.850435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.848290 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:44.854438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.853123 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:44.854438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.853175 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:44.891440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.877949 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:44.943495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:44.943323 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:45.044097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.043657 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 03:06:45.044097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.043733 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 03:06:45.259496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.258865 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:06:45.259496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.258929 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:06:45.450797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.450287 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:06:45.651725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.651608 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:45.660660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.660624 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" event={"ID":"60edd9f0-94a4-4219-9bc9-f73a1044af7f","Type":"ContainerStarted","Data":"22a9a5249c63e5715a461bfb988616052d19c0a4ddca84d396e03d4e1de1121a"} Oct 31 03:06:45.750316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.750276 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 31 03:06:45.750527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:45.750341 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 31 03:06:45.944749 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:06:45.944701 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:06:46.062562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.062518 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:46.064261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.064234 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:06:46.233589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.233513 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:46.234596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.234577 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:06:46.437861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.437831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:46.667637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.667591 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" event={"ID":"60edd9f0-94a4-4219-9bc9-f73a1044af7f","Type":"ContainerStarted","Data":"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b"} Oct 31 03:06:46.686920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.686872 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podStartSLOduration=2.686857872 podStartE2EDuration="2.686857872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:46.684837533 +0000 UTC m=+684.539854675" watchObservedRunningTime="2026-10-31 03:06:46.686857872 +0000 UTC m=+684.541875013" Oct 31 03:06:46.754491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.754463 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:46.755448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.755402 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:06:46.886435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:46.886386 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:47.343880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:47.343845 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:47.453031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:47.452999 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:48.150486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:48.150451 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:48.163770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:48.163740 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:48.425526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:48.425409 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:48.691078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:48.690989 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:06:49.221121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:49.221080 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:50.122263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:50.122225 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:50.553186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:50.553152 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:51.055238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.055203 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:51.329344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.329263 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:06:51.333538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.333518 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.338607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.338578 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"kube-root-ca.crt\"" Oct 31 03:06:51.338736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.338686 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"simulator-w3-99bc97-kserve-self-signed-certs\"" Oct 31 03:06:51.341315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.341291 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"openshift-service-ca.crt\"" Oct 31 03:06:51.341433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.341331 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"default-dockercfg-2rmv8\"" Oct 31 03:06:51.344776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.344752 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:06:51.409292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.409258 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:51.485964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.485927 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.486152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.485979 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.486152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.486135 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.486262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.486170 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.486301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.486277 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.486345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.486314 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spshc\" (UniqueName: \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587212 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587247 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587271 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587392 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spshc\" (UniqueName: \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587530 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587590 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587691 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587734 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.587952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.587933 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.589654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.589632 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.589907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.589890 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.599011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.598991 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spshc\" (UniqueName: \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\") pod \"simulator-w3-99bc97-kserve-79f978b49-d62mg\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.646875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.646842 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:51.925291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.925256 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:06:51.925409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:51.925311 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:51.929864 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:06:51.929837 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69f434bd_237c_4c9f_aa3d_af3599c7feeb.slice/crio-07628ae33697e0aa1cd0d337acc6d98042dfdb71ff8113cc80208d2ac3f419d7 WatchSource:0}: Error finding container 07628ae33697e0aa1cd0d337acc6d98042dfdb71ff8113cc80208d2ac3f419d7: Status 404 returned error can't find the container with id 07628ae33697e0aa1cd0d337acc6d98042dfdb71ff8113cc80208d2ac3f419d7 Oct 31 03:06:52.431346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:52.431307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:52.718757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:52.718674 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" event={"ID":"69f434bd-237c-4c9f-aa3d-af3599c7feeb","Type":"ContainerStarted","Data":"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a"} Oct 31 03:06:52.718757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:52.718715 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" event={"ID":"69f434bd-237c-4c9f-aa3d-af3599c7feeb","Type":"ContainerStarted","Data":"07628ae33697e0aa1cd0d337acc6d98042dfdb71ff8113cc80208d2ac3f419d7"} Oct 31 03:06:52.739943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:52.739890 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podStartSLOduration=1.739877067 podStartE2EDuration="1.739877067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:52.737934798 +0000 UTC m=+690.592951938" watchObservedRunningTime="2026-10-31 03:06:52.739877067 +0000 UTC m=+690.594894208" Oct 31 03:06:52.748197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:52.748171 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:53.127225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:53.127145 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:53.266461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:53.266380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:53.395659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:53.395570 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:53.647324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:53.647240 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:06:53.940101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:53.940055 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:54.638995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:54.638960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:54.886595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:54.886566 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:54.899354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:54.899289 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:55.336733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:55.336693 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:55.746038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:55.746011 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:06:55.976497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:55.976462 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:06:56.039601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:56.039517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:56.743025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:56.742953 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:57.015472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:57.015322 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:57.833584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:57.833532 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:58.624868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:58.624809 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:59.336595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.336559 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:59.541596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.541564 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:06:59.545787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.545764 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.553513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.553472 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"default-dockercfg-6w94r\"" Oct 31 03:06:59.553631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.553527 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"simulator-w1-924ee5-kserve-self-signed-certs\"" Oct 31 03:06:59.553849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.553815 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"openshift-service-ca.crt\"" Oct 31 03:06:59.554181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.554159 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"kube-root-ca.crt\"" Oct 31 03:06:59.556437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.556398 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:06:59.621916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.621838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:06:59.680629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680593 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlp2w\" (UniqueName: \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.681018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680673 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.681018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680736 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.681018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680811 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.681018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680853 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.681018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.680910 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782461 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782540 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782571 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782691 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782729 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782819 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlp2w\" (UniqueName: \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.782969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.782946 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.783108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.783069 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.783191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.783172 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.784821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.784798 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.785098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.785078 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.794098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.794073 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlp2w\" (UniqueName: \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\") pod \"simulator-w1-924ee5-kserve-678b85cb94-xzlf5\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:06:59.859124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:06:59.859086 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:07:00.136892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.135962 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:00.144846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.144814 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:07:00.147987 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:00.147950 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70af976b_5785_4f55_ba46_c652c94a56c0.slice/crio-41334d13b93169e695a99a1584c16316357c04bcaae189a029f54efdf9c05fe4 WatchSource:0}: Error finding container 41334d13b93169e695a99a1584c16316357c04bcaae189a029f54efdf9c05fe4: Status 404 returned error can't find the container with id 41334d13b93169e695a99a1584c16316357c04bcaae189a029f54efdf9c05fe4 Oct 31 03:07:00.764638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.764603 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" event={"ID":"70af976b-5785-4f55-ba46-c652c94a56c0","Type":"ContainerStarted","Data":"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519"} Oct 31 03:07:00.764638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.764646 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" event={"ID":"70af976b-5785-4f55-ba46-c652c94a56c0","Type":"ContainerStarted","Data":"41334d13b93169e695a99a1584c16316357c04bcaae189a029f54efdf9c05fe4"} Oct 31 03:07:00.784618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.784570 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podStartSLOduration=1.7845568429999998 podStartE2EDuration="1.784556843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:00.782672368 +0000 UTC m=+698.637689486" watchObservedRunningTime="2026-10-31 03:07:00.784556843 +0000 UTC m=+698.639573984" Oct 31 03:07:00.859895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.859859 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:07:00.911307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:00.911271 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:01.647851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:01.647802 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:07:01.661056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:01.661030 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:07:01.725997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:01.725965 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:01.786792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:01.786767 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:07:02.432857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:02.432819 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:02.561381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:02.561349 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:03.218275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:03.218232 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:03.989580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:03.989544 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:04.829191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:04.829139 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:05.515427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:05.515384 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:06.224006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:06.223973 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:06.925826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:06.925784 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:07:07.015430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:07.015381 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:07.559056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:07.559016 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:07.873438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:07.873303 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:08.533713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:08.533683 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:08.665649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:08.665619 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:09.334545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:09.334511 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:09.505439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:09.505383 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:09.859641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:09.859600 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:07:09.872551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:09.872524 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:07:10.267213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:10.267179 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:10.840068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:10.840042 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:07:11.134640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:11.134556 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:11.935547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:11.935512 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:12.072229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:12.072189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:12.630951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:12.630918 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:13.413478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:13.413440 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:14.226271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:14.226226 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:14.923382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:14.923345 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:15.719008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:15.718977 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:15.935085 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:07:15.935052 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:07:16.436486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:16.436393 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:17.322604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:17.322559 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:18.120840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.120792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:18.509236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.509207 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:07:18.513883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.513859 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.521918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.521447 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"openshift-service-ca.crt\"" Oct 31 03:07:18.522191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.522172 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"premium-w4-bebea4-kserve-self-signed-certs\"" Oct 31 03:07:18.522340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.522325 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"default-dockercfg-mgsmr\"" Oct 31 03:07:18.522640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.522622 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"kube-root-ca.crt\"" Oct 31 03:07:18.525108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.525085 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:07:18.592657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592623 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.592819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592677 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.592819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592751 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.592951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592857 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.592951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592901 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.593016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.592962 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj96j\" (UniqueName: \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.597356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.597335 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:18.694267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694226 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694273 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694318 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj96j\" (UniqueName: \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694389 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694465 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694501 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694737 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694805 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.694897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.694848 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.696554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.696530 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.696848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.696831 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.705638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.705614 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj96j\" (UniqueName: \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\") pod \"premium-w4-bebea4-kserve-75f9cbb589-72qzd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.827816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.827727 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:18.925387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:18.925355 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:19.107198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.107170 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:07:19.109516 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:19.109486 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80327f0f_af3d_4a66_b1c1_a05cb396b8fd.slice/crio-12a7f062dbc1583fcdfeb164cfe92231fd87d28c6dc1505cd5b8fac008436fbe WatchSource:0}: Error finding container 12a7f062dbc1583fcdfeb164cfe92231fd87d28c6dc1505cd5b8fac008436fbe: Status 404 returned error can't find the container with id 12a7f062dbc1583fcdfeb164cfe92231fd87d28c6dc1505cd5b8fac008436fbe Oct 31 03:07:19.639879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.639825 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:19.842234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.842140 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:19.886855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.886815 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" event={"ID":"80327f0f-af3d-4a66-b1c1-a05cb396b8fd","Type":"ContainerStarted","Data":"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5"} Oct 31 03:07:19.887092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.886869 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" event={"ID":"80327f0f-af3d-4a66-b1c1-a05cb396b8fd","Type":"ContainerStarted","Data":"12a7f062dbc1583fcdfeb164cfe92231fd87d28c6dc1505cd5b8fac008436fbe"} Oct 31 03:07:19.919736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:19.919679 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podStartSLOduration=1.919663446 podStartE2EDuration="1.919663446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:19.918612233 +0000 UTC m=+717.773629366" watchObservedRunningTime="2026-10-31 03:07:19.919663446 +0000 UTC m=+717.774680587" Oct 31 03:07:20.144561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:20.144483 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:07:20.519474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:20.519070 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:20.827965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:20.827882 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:21.267359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:21.267312 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:22.022793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:22.022755 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:22.712059 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:22.712014 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:23.084559 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:23.084454 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:23.525644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:23.525608 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:24.399584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:24.399550 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:25.108635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:25.108598 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:25.908089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:25.908056 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:26.501208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.501176 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:07:26.506164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.506144 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.520159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.520134 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-924ee5\"/\"premium-w1-924ee5-kserve-self-signed-certs\"" Oct 31 03:07:26.535553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.535523 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:07:26.591228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591196 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.591228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591229 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j9n\" (UniqueName: \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.591407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591313 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.591407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591352 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.591407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591376 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.591612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.591589 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.597670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.597651 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:26.692176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692142 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692197 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692240 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692382 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692474 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692500 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j9n\" (UniqueName: \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692692 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692728 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.692846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.692827 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.694533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.694506 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.694778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.694758 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.704087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.704066 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j9n\" (UniqueName: \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\") pod \"premium-w1-924ee5-kserve-6d7988bd59-999rh\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:26.809878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.809785 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:26.818802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:26.818771 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:27.095244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:27.095223 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:07:27.098309 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:27.098280 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b702eeb_fa9e_460b_8c81_1feeea666d69.slice/crio-3bf535ab9816302158788c1dd5341c3796f40a192f81e53c5cdf0d0b137c8f9a WatchSource:0}: Error finding container 3bf535ab9816302158788c1dd5341c3796f40a192f81e53c5cdf0d0b137c8f9a: Status 404 returned error can't find the container with id 3bf535ab9816302158788c1dd5341c3796f40a192f81e53c5cdf0d0b137c8f9a Oct 31 03:07:27.624403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:27.624363 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:27.940422 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:27.940385 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" event={"ID":"4b702eeb-fa9e-460b-8c81-1feeea666d69","Type":"ContainerStarted","Data":"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d"} Oct 31 03:07:27.940620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:27.940450 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" event={"ID":"4b702eeb-fa9e-460b-8c81-1feeea666d69","Type":"ContainerStarted","Data":"3bf535ab9816302158788c1dd5341c3796f40a192f81e53c5cdf0d0b137c8f9a"} Oct 31 03:07:27.959239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:27.959177 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podStartSLOduration=1.959164616 podStartE2EDuration="1.959164616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:27.958795985 +0000 UTC m=+725.813813126" watchObservedRunningTime="2026-10-31 03:07:27.959164616 +0000 UTC m=+725.814181755" Oct 31 03:07:28.451437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.430849 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:28.583476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.582674 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:28.821491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.820486 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:28.831485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.830697 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:28.888954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.888700 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:28.981794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:28.981765 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:07:29.224588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:29.224549 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:30.120790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:30.120749 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:30.846874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:30.846841 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:31.818513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:31.818447 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:32.632751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:32.632698 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:33.390723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:33.390691 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:34.139096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:34.139060 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:34.912227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:34.912192 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:35.841233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:35.841197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:36.532600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:36.532559 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:36.819642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:36.819543 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:36.833110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:36.833083 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:37.009769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:37.009738 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:07:37.585668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:37.585627 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:38.249021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:38.248984 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:07:38.320698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:38.320629 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:38.625496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:38.625402 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:39.644956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:39.644922 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:40.440696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:40.440660 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:41.419438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:41.418145 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:42.474211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:42.474167 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:42.982424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:42.982379 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:07:42.986859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:42.986835 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:42.991361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:42.991338 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-85cb59-openshift-default-dockercfg-fr8qt\"" Oct 31 03:07:42.999580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:42.999538 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:07:43.148839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.148801 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.148846 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.148942 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.148996 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.149043 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppkn\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.149074 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.149126 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.149156 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.149232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.149184 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.250905 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.250960 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251032 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251063 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251126 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppkn\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251158 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251188 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251228 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251268 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251716 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251781 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.251844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.251804 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.252044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.252018 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.252113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.252047 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.253609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.253585 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.253874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.253852 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.262042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.262014 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.262169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.262083 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppkn\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.303956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.303923 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:43.498649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.498608 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:43.591616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.591594 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:07:43.594582 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:43.594549 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f6ada_3a66_45f1_acd8_64f2729f4ba2.slice/crio-f6d43b27ab5489842226d167bde2b38c414d8ffad96613b8e9466ac371dd60d3 WatchSource:0}: Error finding container f6d43b27ab5489842226d167bde2b38c414d8ffad96613b8e9466ac371dd60d3: Status 404 returned error can't find the container with id f6d43b27ab5489842226d167bde2b38c414d8ffad96613b8e9466ac371dd60d3 Oct 31 03:07:43.597253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.597209 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:43.597346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.597300 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:43.597388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:43.597340 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:44.045713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.044629 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" event={"ID":"405f6ada-3a66-45f1-acd8-64f2729f4ba2","Type":"ContainerStarted","Data":"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52"} Oct 31 03:07:44.045713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.044722 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" event={"ID":"405f6ada-3a66-45f1-acd8-64f2729f4ba2","Type":"ContainerStarted","Data":"f6d43b27ab5489842226d167bde2b38c414d8ffad96613b8e9466ac371dd60d3"} Oct 31 03:07:44.070964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.070909 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podStartSLOduration=2.07089364 podStartE2EDuration="2.07089364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:44.067699244 +0000 UTC m=+741.922716398" watchObservedRunningTime="2026-10-31 03:07:44.07089364 +0000 UTC m=+741.925910780" Oct 31 03:07:44.228439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.226749 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:44.304853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.304762 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:44.306843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.306817 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:44.306903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:44.306884 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:45.105203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:45.105166 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:45.305002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:45.304965 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:45.305176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:45.305034 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:45.910665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:45.910633 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:45.937060 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:07:45.937024 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:07:46.305284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:46.305193 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:46.305284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:46.305270 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:46.706875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:46.706837 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:47.304936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:47.304900 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:47.305212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:47.304997 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:47.431564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:47.431523 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:48.125352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:48.125313 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:48.304955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:48.304917 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:48.305141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:48.304987 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:48.894526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:48.894478 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:49.304974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:49.304934 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 03:07:49.305254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:49.305222 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 03:07:49.643834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:49.643692 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:50.523542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:50.523495 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:51.305554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.305501 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:07:51.305768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.305588 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:07:51.324599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.324556 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:51.325264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.325232 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:51.326055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.326031 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:07:51.541183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:51.541147 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:52.436040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:52.436003 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:53.462030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:53.461970 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:54.315994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:54.315955 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:55.126786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:55.126712 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:56.028625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:56.027953 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:56.736289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:56.736250 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:57.416115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.416075 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:57.792381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.792303 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:07:57.804198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.804168 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.807728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.807705 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-924ee5-openshift-default-dockercfg-w4fmh\"" Oct 31 03:07:57.810206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.810181 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:07:57.862834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.862799 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.862988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.862848 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.862988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.862949 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863063 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863005 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863075 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdpt\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863109 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863163 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863236 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.863290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.863278 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.963967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.963932 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.963994 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964024 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964049 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964073 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964095 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964137 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964202 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964246 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdpt\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964520 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964619 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964770 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964826 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.964900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.964874 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.967005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.966978 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.967174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.967156 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.975086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.975062 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdpt\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:57.975352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:57.975328 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:58.117174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.117076 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:58.120365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.120337 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:07:58.193859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.191973 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:07:58.200230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.199827 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.208389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.208030 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d8daf-openshift-default-dockercfg-lm6g8\"" Oct 31 03:07:58.214471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.211627 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:07:58.267536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267506 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267642 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267674 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267717 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267766 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267802 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267833 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267910 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rvj\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.267987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.267973 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369087 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369121 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369142 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369163 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369196 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369230 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369252 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rvj\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369314 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369355 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369563 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.369947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.369921 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.370133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.370105 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.370245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.370130 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.370245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.370170 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.371883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.371862 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.372155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.372136 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.380721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.380696 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.380921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.380894 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rvj\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.407748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.407721 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:07:58.410686 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:58.410657 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278bcb1f_859e_46eb_9cf7_8839fb4376cf.slice/crio-17f3748c443e75e3ffad5ac64e3126b11d7b55ea6b7b87a1d4edac057d0e41a1 WatchSource:0}: Error finding container 17f3748c443e75e3ffad5ac64e3126b11d7b55ea6b7b87a1d4edac057d0e41a1: Status 404 returned error can't find the container with id 17f3748c443e75e3ffad5ac64e3126b11d7b55ea6b7b87a1d4edac057d0e41a1 Oct 31 03:07:58.413056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.413026 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:58.413136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.413090 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:58.413191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.413122 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:58.520571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.520531 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:58.807702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.807666 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:07:58.810808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.810763 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:07:58.812609 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:07:58.812578 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfccb95ac_a041_4acf_a984_d2cda2d08cc3.slice/crio-544f4ba23f51aff157c2039419743075f1580975b12382d3f49ef6219483e93c WatchSource:0}: Error finding container 544f4ba23f51aff157c2039419743075f1580975b12382d3f49ef6219483e93c: Status 404 returned error can't find the container with id 544f4ba23f51aff157c2039419743075f1580975b12382d3f49ef6219483e93c Oct 31 03:07:58.815988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.815937 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:58.816109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.816061 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:58.816205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:58.816114 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:07:59.206831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.206797 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" event={"ID":"278bcb1f-859e-46eb-9cf7-8839fb4376cf","Type":"ContainerStarted","Data":"853c3cfa2699256e140e4f54c4005e379ee3a3fe7b7f6822abb491da4387a596"} Oct 31 03:07:59.207010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.206850 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" event={"ID":"278bcb1f-859e-46eb-9cf7-8839fb4376cf","Type":"ContainerStarted","Data":"17f3748c443e75e3ffad5ac64e3126b11d7b55ea6b7b87a1d4edac057d0e41a1"} Oct 31 03:07:59.208858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.208829 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" event={"ID":"fccb95ac-a041-4acf-a984-d2cda2d08cc3","Type":"ContainerStarted","Data":"537ffddc5550ffddf3055cbc77492f3962a4d776e6d8103d4adc100d787feff8"} Oct 31 03:07:59.208982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.208881 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" event={"ID":"fccb95ac-a041-4acf-a984-d2cda2d08cc3","Type":"ContainerStarted","Data":"544f4ba23f51aff157c2039419743075f1580975b12382d3f49ef6219483e93c"} Oct 31 03:07:59.229315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.229257 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podStartSLOduration=2.22923804 podStartE2EDuration="2.22923804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:59.227973466 +0000 UTC m=+757.082990609" watchObservedRunningTime="2026-10-31 03:07:59.22923804 +0000 UTC m=+757.084255180" Oct 31 03:07:59.251690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.251591 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podStartSLOduration=1.251576013 podStartE2EDuration="1.251576013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:59.248836347 +0000 UTC m=+757.103853490" watchObservedRunningTime="2026-10-31 03:07:59.251576013 +0000 UTC m=+757.106593154" Oct 31 03:07:59.521397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.521201 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:07:59.523167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.523131 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:07:59.523296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.523178 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:07:59.631928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:07:59.631656 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:00.120817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.120782 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:08:00.122228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.122199 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:08:00.122370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.122252 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:08:00.441439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.438258 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:00.521903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.521859 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:08:00.522083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:00.521911 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:08:01.121476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.121408 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:08:01.121878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.121503 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:08:01.240666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.238655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:01.522010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.521971 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:08:01.522197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.522031 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:08:01.939739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:01.939700 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:02.121498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:02.121458 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:08:02.121899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:02.121518 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:08:02.520994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:02.520961 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:08:02.521175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:02.521026 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:08:02.720300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:02.720260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:03.122057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:03.122014 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:08:03.122597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:03.122082 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:08:03.523440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:03.522846 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:08:03.523440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:03.522927 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:08:03.528351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:03.528097 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:04.122927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.122839 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:08:04.123564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.123520 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:08:04.255890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.250648 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:04.521339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.521295 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:08:04.521524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.521366 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:08:04.646640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:04.646607 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:08:05.025200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.025160 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:05.373314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.373227 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:08:05.375149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.375125 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:08:05.376480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.376403 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:08:05.781432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.781238 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:08:05.782158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.782048 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:08:05.783765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.783742 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:08:05.907268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:05.907218 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:06.731632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.731583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:06.867757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.867721 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc"] Oct 31 03:08:06.872731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.872714 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:06.876260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.876234 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:06.878588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.878566 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:06.882905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.882876 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc"] Oct 31 03:08:06.951905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.951855 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s762f\" (UniqueName: \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:06.952086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.952027 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:06.952086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:06.952080 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.053512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.053400 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s762f\" (UniqueName: \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.053667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.053561 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.053667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.053611 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.054235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.054208 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.055955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.055932 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.065484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.065463 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s762f\" (UniqueName: \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\") pod \"maas-api-revoke-sub-dc7e8c13-r8wtc\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.188650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.188612 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:07.463536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.463513 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc"] Oct 31 03:08:07.465480 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:07.465451 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43901272_4ba1_4a18_b2a2_1f22725be21c.slice/crio-505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff WatchSource:0}: Error finding container 505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff: Status 404 returned error can't find the container with id 505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff Oct 31 03:08:07.566353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.566314 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:07.707708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:07.707670 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:08.273731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:08.273688 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" event={"ID":"43901272-4ba1-4a18-b2a2-1f22725be21c","Type":"ContainerStarted","Data":"505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff"} Oct 31 03:08:08.435241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:08.435180 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:08.599877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:08.599768 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:09.137903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:09.137838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:09.828877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:09.828819 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:10.152852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.152818 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8"] Oct 31 03:08:10.161919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.161893 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.167161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.167134 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8"] Oct 31 03:08:10.194111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.194077 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.194273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.194253 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2nm\" (UniqueName: \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.194343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.194327 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.295331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.295297 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.295509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.295396 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.295565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.295537 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2nm\" (UniqueName: \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.296070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.296038 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.297991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.297964 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.307189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.307161 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2nm\" (UniqueName: \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\") pod \"maas-api-revoke-sub-8e666809-cdkc8\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.480392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.480304 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:10.531126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.531094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:10.783835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:10.783698 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8"] Oct 31 03:08:10.785720 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:10.785692 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e465a17_e093_4986_8b56_5005c318a957.slice/crio-8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a WatchSource:0}: Error finding container 8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a: Status 404 returned error can't find the container with id 8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a Oct 31 03:08:11.241338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.241228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:11.295142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.295109 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" event={"ID":"43901272-4ba1-4a18-b2a2-1f22725be21c","Type":"ContainerStarted","Data":"bcb0ee5fd51b8ce3bc36c8c9459886a8effc5da414e4d27236dc25732ccc0c17"} Oct 31 03:08:11.296726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.296696 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" event={"ID":"5e465a17-e093-4986-8b56-5005c318a957","Type":"ContainerStarted","Data":"6e30fc1ed9640cc2711d833f0c33751367d97f119c215ecffc33ad1e01ce5e01"} Oct 31 03:08:11.296857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.296750 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" event={"ID":"5e465a17-e093-4986-8b56-5005c318a957","Type":"ContainerStarted","Data":"8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a"} Oct 31 03:08:11.314954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.314900 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" podStartSLOduration=2.074792985 podStartE2EDuration="5.314884486s" totalImagesPullingTime="3.240091501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:11.312533155 +0000 UTC m=+769.167550296" watchObservedRunningTime="2026-10-31 03:08:11.314884486 +0000 UTC m=+769.169901627" Oct 31 03:08:11.330098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.330014 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" podStartSLOduration=1.329998877 podStartE2EDuration="1.329998877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:11.328443829 +0000 UTC m=+769.183460964" watchObservedRunningTime="2026-10-31 03:08:11.329998877 +0000 UTC m=+769.185016017" Oct 31 03:08:11.395798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:11.389503 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:12.009964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.009932 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:12.304064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.303957 4556 generic.go:356] "Generic (PLEG): container finished" podID="43901272-4ba1-4a18-b2a2-1f22725be21c" containerID="bcb0ee5fd51b8ce3bc36c8c9459886a8effc5da414e4d27236dc25732ccc0c17" exitCode=0 Oct 31 03:08:12.304541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.304375 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" event={"ID":"43901272-4ba1-4a18-b2a2-1f22725be21c","Type":"ContainerDied","Data":"bcb0ee5fd51b8ce3bc36c8c9459886a8effc5da414e4d27236dc25732ccc0c17"} Oct 31 03:08:12.306442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.306391 4556 generic.go:356] "Generic (PLEG): container finished" podID="5e465a17-e093-4986-8b56-5005c318a957" containerID="6e30fc1ed9640cc2711d833f0c33751367d97f119c215ecffc33ad1e01ce5e01" exitCode=0 Oct 31 03:08:12.306912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.306454 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" event={"ID":"5e465a17-e093-4986-8b56-5005c318a957","Type":"ContainerDied","Data":"6e30fc1ed9640cc2711d833f0c33751367d97f119c215ecffc33ad1e01ce5e01"} Oct 31 03:08:12.811783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:12.811730 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:13.465730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.465703 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:13.499406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.499380 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:13.542093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542060 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\" (UniqueName: \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\") pod \"43901272-4ba1-4a18-b2a2-1f22725be21c\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " Oct 31 03:08:13.542093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542093 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\") pod \"43901272-4ba1-4a18-b2a2-1f22725be21c\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " Oct 31 03:08:13.542296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542182 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\") pod \"5e465a17-e093-4986-8b56-5005c318a957\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " Oct 31 03:08:13.542296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542207 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\") pod \"5e465a17-e093-4986-8b56-5005c318a957\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " Oct 31 03:08:13.542394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542317 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\") pod \"43901272-4ba1-4a18-b2a2-1f22725be21c\" (UID: \"43901272-4ba1-4a18-b2a2-1f22725be21c\") " Oct 31 03:08:13.542394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542353 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\" (UniqueName: \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\") pod \"5e465a17-e093-4986-8b56-5005c318a957\" (UID: \"5e465a17-e093-4986-8b56-5005c318a957\") " Oct 31 03:08:13.542880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542678 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca" pod "5e465a17-e093-4986-8b56-5005c318a957" (UID: "5e465a17-e093-4986-8b56-5005c318a957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:13.542880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542830 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca" pod "43901272-4ba1-4a18-b2a2-1f22725be21c" (UID: "43901272-4ba1-4a18-b2a2-1f22725be21c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:13.543073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542970 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:13.543073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.542991 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e465a17-e093-4986-8b56-5005c318a957-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:13.544662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.544632 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca" pod "43901272-4ba1-4a18-b2a2-1f22725be21c" (UID: "43901272-4ba1-4a18-b2a2-1f22725be21c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:13.544861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.544834 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm" pod "5e465a17-e093-4986-8b56-5005c318a957" (UID: "5e465a17-e093-4986-8b56-5005c318a957"). InnerVolumeSpecName "kube-api-access-pl2nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:13.545065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.545045 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca" pod "5e465a17-e093-4986-8b56-5005c318a957" (UID: "5e465a17-e093-4986-8b56-5005c318a957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:13.545174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.545154 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f" pod "43901272-4ba1-4a18-b2a2-1f22725be21c" (UID: "43901272-4ba1-4a18-b2a2-1f22725be21c"). InnerVolumeSpecName "kube-api-access-s762f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:13.633514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.633430 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:13.643933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.643903 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s762f\" (UniqueName: \"kubernetes.io/projected/43901272-4ba1-4a18-b2a2-1f22725be21c-kube-api-access-s762f\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:13.643933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.643934 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e465a17-e093-4986-8b56-5005c318a957-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:13.644087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.643948 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901272-4ba1-4a18-b2a2-1f22725be21c-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:13.644087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:13.643962 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2nm\" (UniqueName: \"kubernetes.io/projected/5e465a17-e093-4986-8b56-5005c318a957-kube-api-access-pl2nm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:14.318661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.318633 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" Oct 31 03:08:14.318833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.318634 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc" event={"ID":"43901272-4ba1-4a18-b2a2-1f22725be21c","Type":"ContainerDied","Data":"505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff"} Oct 31 03:08:14.318833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.318746 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505b8feb9cb010a9980a8099c966dfed4bdb3ae4a5d02978dd58fae19f8941ff" Oct 31 03:08:14.320052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.320028 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" event={"ID":"5e465a17-e093-4986-8b56-5005c318a957","Type":"ContainerDied","Data":"8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a"} Oct 31 03:08:14.320052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.320051 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8" Oct 31 03:08:14.320190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.320071 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1ddc79f6d012a7ce36f32c46e934619e18837d4c7c6db1f8148725ca7ed26a" Oct 31 03:08:14.333110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:14.333079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:15.073839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:15.073798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:15.936003 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:08:15.935965 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:08:16.021795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:16.020502 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:16.739835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:16.739778 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:17.436307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:17.436272 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:18.135769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.135732 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:18.163581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.163538 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd"] Oct 31 03:08:18.164375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164351 4556 cpu_manager.go:414] "containerMap: removing container" podUID="5e465a17-e093-4986-8b56-5005c318a957" containerName="revoke-keys" Oct 31 03:08:18.164375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164375 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e465a17-e093-4986-8b56-5005c318a957" containerName="revoke-keys" Oct 31 03:08:18.164618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164402 4556 cpu_manager.go:414] "containerMap: removing container" podUID="43901272-4ba1-4a18-b2a2-1f22725be21c" containerName="revoke-keys" Oct 31 03:08:18.164618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164425 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43901272-4ba1-4a18-b2a2-1f22725be21c" containerName="revoke-keys" Oct 31 03:08:18.164618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164547 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e465a17-e093-4986-8b56-5005c318a957" containerName="revoke-keys" Oct 31 03:08:18.164618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.164564 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="43901272-4ba1-4a18-b2a2-1f22725be21c" containerName="revoke-keys" Oct 31 03:08:18.169362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.169336 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.172737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.172713 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:18.174630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.174606 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:18.177068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.177044 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd"] Oct 31 03:08:18.262744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.262702 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.265540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.265511 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxbg\" (UniqueName: \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.265871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.265852 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.367154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.367113 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.367322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.367183 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxbg\" (UniqueName: \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.367383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.367321 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.368010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.367987 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.369634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.369615 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.379065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.379034 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxbg\" (UniqueName: \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\") pod \"maas-api-revoke-sub-b2c24efc-4n4cd\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.487371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.487332 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:18.773181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.773146 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd"] Oct 31 03:08:18.774514 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:18.774481 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a00190c_28a7_4956_b553_b81f6b5e5c96.slice/crio-422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226 WatchSource:0}: Error finding container 422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226: Status 404 returned error can't find the container with id 422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226 Oct 31 03:08:18.930508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:18.930474 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:19.353968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.353930 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" event={"ID":"0a00190c-28a7-4956-b553-b81f6b5e5c96","Type":"ContainerStarted","Data":"fbbbde3a2e7aa95c9b581b7ef614b178ca0c0da1a18accbd09b51cec081b47ee"} Oct 31 03:08:19.354490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.353996 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" event={"ID":"0a00190c-28a7-4956-b553-b81f6b5e5c96","Type":"ContainerStarted","Data":"422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226"} Oct 31 03:08:19.374616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.374551 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" podStartSLOduration=1.374524578 podStartE2EDuration="1.374524578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:19.371300409 +0000 UTC m=+777.226317551" watchObservedRunningTime="2026-10-31 03:08:19.374524578 +0000 UTC m=+777.229541724" Oct 31 03:08:19.643833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.643791 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74"] Oct 31 03:08:19.648372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.648345 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.660850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.660816 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74"] Oct 31 03:08:19.789232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.789194 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.789407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.789253 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfzw\" (UniqueName: \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.789492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.789406 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.892475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.891025 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.892475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.891099 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfzw\" (UniqueName: \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.892475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.891180 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.892475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.892163 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.895173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.895149 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.905145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.905120 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfzw\" (UniqueName: \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\") pod \"maas-api-revoke-sub-c3c98028-qfd74\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:19.924275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.924236 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:19.965010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:19.964974 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:20.003513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.003474 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj"] Oct 31 03:08:20.014436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.011645 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.018433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.017773 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj"] Oct 31 03:08:20.095231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.095194 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9ml\" (UniqueName: \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.095401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.095335 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.095617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.095594 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.196335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.196296 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9ml\" (UniqueName: \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.196530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.196363 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.196530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.196520 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.197062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.197034 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.198843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.198825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.207567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.207544 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9ml\" (UniqueName: \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\") pod \"maas-api-revoke-sub-2d6858d9-sb9xj\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.282290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.282259 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74"] Oct 31 03:08:20.283282 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:20.283246 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6add1fc1_c5a8_4308_bc71_1b8e7cccf270.slice/crio-3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe WatchSource:0}: Error finding container 3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe: Status 404 returned error can't find the container with id 3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe Oct 31 03:08:20.351802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.351762 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:20.375187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.375157 4556 generic.go:356] "Generic (PLEG): container finished" podID="0a00190c-28a7-4956-b553-b81f6b5e5c96" containerID="fbbbde3a2e7aa95c9b581b7ef614b178ca0c0da1a18accbd09b51cec081b47ee" exitCode=0 Oct 31 03:08:20.375653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.375231 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" event={"ID":"0a00190c-28a7-4956-b553-b81f6b5e5c96","Type":"ContainerDied","Data":"fbbbde3a2e7aa95c9b581b7ef614b178ca0c0da1a18accbd09b51cec081b47ee"} Oct 31 03:08:20.377025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.376992 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" event={"ID":"6add1fc1-c5a8-4308-bc71-1b8e7cccf270","Type":"ContainerStarted","Data":"3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe"} Oct 31 03:08:20.727362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.727267 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:20.772029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:20.771990 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj"] Oct 31 03:08:21.384454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.384332 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" event={"ID":"6add1fc1-c5a8-4308-bc71-1b8e7cccf270","Type":"ContainerStarted","Data":"8703caeb9626ee705750811bc18aadd9e87e46700c3b14c5333efe96bb27cec9"} Oct 31 03:08:21.386133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.386097 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" event={"ID":"385dcfea-78ec-4f66-b6ff-da6e7aaef53f","Type":"ContainerStarted","Data":"02d5bb494ed434c79600387f23d2ef9e73461b7461412f72f258939d941f0a93"} Oct 31 03:08:21.386238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.386147 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" event={"ID":"385dcfea-78ec-4f66-b6ff-da6e7aaef53f","Type":"ContainerStarted","Data":"a38937c93659b3652a337e7c87909f8c10ec13ad481b0bdbf008260801f1b69d"} Oct 31 03:08:21.404572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.404517 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" podStartSLOduration=2.404501284 podStartE2EDuration="2.404501284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:21.403507644 +0000 UTC m=+779.258524782" watchObservedRunningTime="2026-10-31 03:08:21.404501284 +0000 UTC m=+779.259518446" Oct 31 03:08:21.424381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.424313 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" podStartSLOduration=2.424298385 podStartE2EDuration="2.424298385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:21.420366241 +0000 UTC m=+779.275383382" watchObservedRunningTime="2026-10-31 03:08:21.424298385 +0000 UTC m=+779.279315525" Oct 31 03:08:21.518539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.516229 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:21.562593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.562567 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:21.719807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.719769 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\" (UniqueName: \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\") pod \"0a00190c-28a7-4956-b553-b81f6b5e5c96\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " Oct 31 03:08:21.720013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.719948 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\") pod \"0a00190c-28a7-4956-b553-b81f6b5e5c96\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " Oct 31 03:08:21.720084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.720044 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\") pod \"0a00190c-28a7-4956-b553-b81f6b5e5c96\" (UID: \"0a00190c-28a7-4956-b553-b81f6b5e5c96\") " Oct 31 03:08:21.720406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.720382 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca" pod "0a00190c-28a7-4956-b553-b81f6b5e5c96" (UID: "0a00190c-28a7-4956-b553-b81f6b5e5c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:21.722005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.721981 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca" pod "0a00190c-28a7-4956-b553-b81f6b5e5c96" (UID: "0a00190c-28a7-4956-b553-b81f6b5e5c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:21.722093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.721984 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg" pod "0a00190c-28a7-4956-b553-b81f6b5e5c96" (UID: "0a00190c-28a7-4956-b553-b81f6b5e5c96"). InnerVolumeSpecName "kube-api-access-wqxbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:21.822025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.821982 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:21.822025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.822018 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxbg\" (UniqueName: \"kubernetes.io/projected/0a00190c-28a7-4956-b553-b81f6b5e5c96-kube-api-access-wqxbg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:21.822025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:21.822032 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00190c-28a7-4956-b553-b81f6b5e5c96-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:22.246325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.246253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:22.393627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.393597 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" Oct 31 03:08:22.394048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.393596 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd" event={"ID":"0a00190c-28a7-4956-b553-b81f6b5e5c96","Type":"ContainerDied","Data":"422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226"} Oct 31 03:08:22.394048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.393776 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422c60d0eb4f6a3fb2e4636355895b9b4ac565da78b9f3418c411d0a1e34c226" Oct 31 03:08:22.395316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.395290 4556 generic.go:356] "Generic (PLEG): container finished" podID="6add1fc1-c5a8-4308-bc71-1b8e7cccf270" containerID="8703caeb9626ee705750811bc18aadd9e87e46700c3b14c5333efe96bb27cec9" exitCode=0 Oct 31 03:08:22.395489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.395370 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" event={"ID":"6add1fc1-c5a8-4308-bc71-1b8e7cccf270","Type":"ContainerDied","Data":"8703caeb9626ee705750811bc18aadd9e87e46700c3b14c5333efe96bb27cec9"} Oct 31 03:08:22.397075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.397042 4556 generic.go:356] "Generic (PLEG): container finished" podID="385dcfea-78ec-4f66-b6ff-da6e7aaef53f" containerID="02d5bb494ed434c79600387f23d2ef9e73461b7461412f72f258939d941f0a93" exitCode=0 Oct 31 03:08:22.397185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.397091 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" event={"ID":"385dcfea-78ec-4f66-b6ff-da6e7aaef53f","Type":"ContainerDied","Data":"02d5bb494ed434c79600387f23d2ef9e73461b7461412f72f258939d941f0a93"} Oct 31 03:08:22.947313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:22.945887 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:23.567576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.567552 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:23.605353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.605321 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:23.647182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.647146 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\") pod \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " Oct 31 03:08:23.647333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.647314 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\") pod \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " Oct 31 03:08:23.647390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.647369 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\" (UniqueName: \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\") pod \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\" (UID: \"6add1fc1-c5a8-4308-bc71-1b8e7cccf270\") " Oct 31 03:08:23.647623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.647595 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca" pod "6add1fc1-c5a8-4308-bc71-1b8e7cccf270" (UID: "6add1fc1-c5a8-4308-bc71-1b8e7cccf270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:23.647754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.647728 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:23.649374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.649341 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca" pod "6add1fc1-c5a8-4308-bc71-1b8e7cccf270" (UID: "6add1fc1-c5a8-4308-bc71-1b8e7cccf270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:23.649517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.649384 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw" pod "6add1fc1-c5a8-4308-bc71-1b8e7cccf270" (UID: "6add1fc1-c5a8-4308-bc71-1b8e7cccf270"). InnerVolumeSpecName "kube-api-access-vjfzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:23.747608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.747570 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:23.748574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.748553 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\" (UniqueName: \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\") pod \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " Oct 31 03:08:23.748698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.748656 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\") pod \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " Oct 31 03:08:23.748771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.748705 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\") pod \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\" (UID: \"385dcfea-78ec-4f66-b6ff-da6e7aaef53f\") " Oct 31 03:08:23.749082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.749053 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca" pod "385dcfea-78ec-4f66-b6ff-da6e7aaef53f" (UID: "385dcfea-78ec-4f66-b6ff-da6e7aaef53f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:23.749082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.749079 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfzw\" (UniqueName: \"kubernetes.io/projected/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-kube-api-access-vjfzw\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:23.749240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.749092 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6add1fc1-c5a8-4308-bc71-1b8e7cccf270-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:23.750723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.750698 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca" pod "385dcfea-78ec-4f66-b6ff-da6e7aaef53f" (UID: "385dcfea-78ec-4f66-b6ff-da6e7aaef53f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:23.750815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.750731 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml" pod "385dcfea-78ec-4f66-b6ff-da6e7aaef53f" (UID: "385dcfea-78ec-4f66-b6ff-da6e7aaef53f"). InnerVolumeSpecName "kube-api-access-fj9ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:23.849791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.849759 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:23.849791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.849786 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:23.849791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:23.849798 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9ml\" (UniqueName: \"kubernetes.io/projected/385dcfea-78ec-4f66-b6ff-da6e7aaef53f-kube-api-access-fj9ml\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:24.410783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.410742 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" event={"ID":"6add1fc1-c5a8-4308-bc71-1b8e7cccf270","Type":"ContainerDied","Data":"3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe"} Oct 31 03:08:24.410969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.410809 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2bc8a75f1bc9adbf7621115bd0e5e63012a0d1189949a3121df04c6eba13fe" Oct 31 03:08:24.410969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.410757 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74" Oct 31 03:08:24.412308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.412276 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" Oct 31 03:08:24.412465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.412315 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj" event={"ID":"385dcfea-78ec-4f66-b6ff-da6e7aaef53f","Type":"ContainerDied","Data":"a38937c93659b3652a337e7c87909f8c10ec13ad481b0bdbf008260801f1b69d"} Oct 31 03:08:24.412465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.412360 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a38937c93659b3652a337e7c87909f8c10ec13ad481b0bdbf008260801f1b69d" Oct 31 03:08:24.640877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:24.640840 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:25.340142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:25.340104 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:26.219684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:26.219645 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:26.927386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:26.926909 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:27.730912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.730864 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:27.808149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808113 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:08:27.808712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808692 4556 cpu_manager.go:414] "containerMap: removing container" podUID="385dcfea-78ec-4f66-b6ff-da6e7aaef53f" containerName="revoke-keys" Oct 31 03:08:27.808712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808712 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385dcfea-78ec-4f66-b6ff-da6e7aaef53f" containerName="revoke-keys" Oct 31 03:08:27.808865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808726 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0a00190c-28a7-4956-b553-b81f6b5e5c96" containerName="revoke-keys" Oct 31 03:08:27.808865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808733 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a00190c-28a7-4956-b553-b81f6b5e5c96" containerName="revoke-keys" Oct 31 03:08:27.808865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808751 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6add1fc1-c5a8-4308-bc71-1b8e7cccf270" containerName="revoke-keys" Oct 31 03:08:27.808865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808758 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6add1fc1-c5a8-4308-bc71-1b8e7cccf270" containerName="revoke-keys" Oct 31 03:08:27.809057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808937 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="385dcfea-78ec-4f66-b6ff-da6e7aaef53f" containerName="revoke-keys" Oct 31 03:08:27.809057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808953 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a00190c-28a7-4956-b553-b81f6b5e5c96" containerName="revoke-keys" Oct 31 03:08:27.809057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.808963 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6add1fc1-c5a8-4308-bc71-1b8e7cccf270" containerName="revoke-keys" Oct 31 03:08:27.813543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.813507 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.816873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.816851 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-56afbd-openshift-default-dockercfg-xms2n\"" Oct 31 03:08:27.843438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.824715 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:08:27.919677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919640 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.919849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919684 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.919849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919708 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919880 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919951 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.919989 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.920096 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.920125 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:27.920257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:27.920237 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.020945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.020863 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.020945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.020907 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.020945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.020929 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.020970 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021119 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021160 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021291 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021353 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021386 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021455 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021605 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021740 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021807 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.021897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.021850 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.023543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.023512 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.024160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.024137 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.034027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.033563 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.034027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.033988 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.154968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.154932 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:28.441112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.441084 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:08:28.441836 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:28.441806 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb209d0b1_af15_4b48_b249_397a6b722aef.slice/crio-e3a61d08eb7db995a9f991c867391a8e9cffbc571c662368e78f42c73bfce88b WatchSource:0}: Error finding container e3a61d08eb7db995a9f991c867391a8e9cffbc571c662368e78f42c73bfce88b: Status 404 returned error can't find the container with id e3a61d08eb7db995a9f991c867391a8e9cffbc571c662368e78f42c73bfce88b Oct 31 03:08:28.445088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.445016 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:28.445188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.445141 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:28.445228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.445186 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:28.561756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.561718 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:28.858302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.858224 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v"] Oct 31 03:08:28.862195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.862173 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:28.868324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.868298 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:28.868495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.868296 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:28.873007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:28.872976 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v"] Oct 31 03:08:29.038024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.037977 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctftn\" (UniqueName: \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.038219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.038046 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.038302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.038242 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.139375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.139266 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.139375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.139352 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctftn\" (UniqueName: \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.139375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.139375 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.140084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.140064 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.141813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.141786 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.151895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.151865 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctftn\" (UniqueName: \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\") pod \"maas-api-revoke-sub-4c886043-b6p8v\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.178676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.177350 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:29.412099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.411973 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:29.458316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.458278 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" event={"ID":"b209d0b1-af15-4b48-b249-397a6b722aef","Type":"ContainerStarted","Data":"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8"} Oct 31 03:08:29.458513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.458407 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" event={"ID":"b209d0b1-af15-4b48-b249-397a6b722aef","Type":"ContainerStarted","Data":"e3a61d08eb7db995a9f991c867391a8e9cffbc571c662368e78f42c73bfce88b"} Oct 31 03:08:29.497439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.497375 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v"] Oct 31 03:08:29.500551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.500503 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podStartSLOduration=2.500488807 podStartE2EDuration="2.500488807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:29.494004443 +0000 UTC m=+787.349021585" watchObservedRunningTime="2026-10-31 03:08:29.500488807 +0000 UTC m=+787.355505948" Oct 31 03:08:29.658736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.658697 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6"] Oct 31 03:08:29.663957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.663905 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.689440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.681763 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6"] Oct 31 03:08:29.848511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.848468 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.848717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.848553 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.848782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.848718 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wmd\" (UniqueName: \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.950153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.950053 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wmd\" (UniqueName: \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.950153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.950127 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.950668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.950173 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.950893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.950865 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.952660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.952633 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.961826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.961796 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wmd\" (UniqueName: \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\") pod \"maas-api-revoke-sub-44bc2c48-8xwd6\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:29.994822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:29.994778 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:30.155429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.155365 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:30.156612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.156585 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:30.156763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.156631 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:30.287251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.280125 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:30.313510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.296979 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6"] Oct 31 03:08:30.316127 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:30.316090 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6836a2_3bca_4deb_a368_2da68208bb1e.slice/crio-779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f WatchSource:0}: Error finding container 779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f: Status 404 returned error can't find the container with id 779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f Oct 31 03:08:30.465348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.465306 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" event={"ID":"0c4b62f0-d4a3-43c1-a351-bfa28deebc86","Type":"ContainerStarted","Data":"8a08a4bf42e7958fb91b15989189d9a8bd098f87292ccb70c14d8e0069c98fb8"} Oct 31 03:08:30.465556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.465371 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" event={"ID":"0c4b62f0-d4a3-43c1-a351-bfa28deebc86","Type":"ContainerStarted","Data":"61bd00937daf19cccbeafbfae6c3e0d46ba0885e1b336243b93b253dcec56afc"} Oct 31 03:08:30.466732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.466702 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" event={"ID":"9f6836a2-3bca-4deb-a368-2da68208bb1e","Type":"ContainerStarted","Data":"779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f"} Oct 31 03:08:30.502655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:30.502594 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" podStartSLOduration=2.502579302 podStartE2EDuration="2.502579302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:30.500329504 +0000 UTC m=+788.355346643" watchObservedRunningTime="2026-10-31 03:08:30.502579302 +0000 UTC m=+788.357596442" Oct 31 03:08:31.031146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.031102 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:31.155895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.155858 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:31.156055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.155914 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:31.498952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.498906 4556 generic.go:356] "Generic (PLEG): container finished" podID="9f6836a2-3bca-4deb-a368-2da68208bb1e" containerID="20183d1a6f0f7bdfe1681a4e26394c0b05b53002a2cc0f3d333f506b1c0b885e" exitCode=0 Oct 31 03:08:31.499142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.499043 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" event={"ID":"9f6836a2-3bca-4deb-a368-2da68208bb1e","Type":"ContainerDied","Data":"20183d1a6f0f7bdfe1681a4e26394c0b05b53002a2cc0f3d333f506b1c0b885e"} Oct 31 03:08:31.500953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.500924 4556 generic.go:356] "Generic (PLEG): container finished" podID="0c4b62f0-d4a3-43c1-a351-bfa28deebc86" containerID="8a08a4bf42e7958fb91b15989189d9a8bd098f87292ccb70c14d8e0069c98fb8" exitCode=0 Oct 31 03:08:31.501684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.501659 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" event={"ID":"0c4b62f0-d4a3-43c1-a351-bfa28deebc86","Type":"ContainerDied","Data":"8a08a4bf42e7958fb91b15989189d9a8bd098f87292ccb70c14d8e0069c98fb8"} Oct 31 03:08:31.796550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.796516 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx"] Oct 31 03:08:31.800428 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.800393 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:31.810634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.810398 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx"] Oct 31 03:08:31.987440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.985331 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:32.002440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.997221 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.002440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.997312 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrs2\" (UniqueName: \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.002440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:31.997429 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.098979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.098883 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.098979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.098944 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrs2\" (UniqueName: \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.099391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.099113 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.099668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.099646 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.101596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.101566 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.110859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.110829 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrs2\" (UniqueName: \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\") pod \"maas-api-revoke-sub-6499afd5-px9wx\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.117578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.117548 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:32.156050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.156019 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:32.156256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.156236 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:32.414197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.414167 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx"] Oct 31 03:08:32.415758 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:32.415732 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf49352_b6c8_49f1_9ef6_a6da94185c2a.slice/crio-aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a WatchSource:0}: Error finding container aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a: Status 404 returned error can't find the container with id aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a Oct 31 03:08:32.507941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.507897 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" event={"ID":"baf49352-b6c8-49f1-9ef6-a6da94185c2a","Type":"ContainerStarted","Data":"aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a"} Oct 31 03:08:32.722442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.722389 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:32.813811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.813730 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\") pod \"9f6836a2-3bca-4deb-a368-2da68208bb1e\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " Oct 31 03:08:32.814111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.813882 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\" (UniqueName: \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\") pod \"9f6836a2-3bca-4deb-a368-2da68208bb1e\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " Oct 31 03:08:32.814111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.814070 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\") pod \"9f6836a2-3bca-4deb-a368-2da68208bb1e\" (UID: \"9f6836a2-3bca-4deb-a368-2da68208bb1e\") " Oct 31 03:08:32.814236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.814178 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca" pod "9f6836a2-3bca-4deb-a368-2da68208bb1e" (UID: "9f6836a2-3bca-4deb-a368-2da68208bb1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:32.814615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.814592 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:32.815898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.815869 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd" pod "9f6836a2-3bca-4deb-a368-2da68208bb1e" (UID: "9f6836a2-3bca-4deb-a368-2da68208bb1e"). InnerVolumeSpecName "kube-api-access-s6wmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:32.816710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.816690 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca" pod "9f6836a2-3bca-4deb-a368-2da68208bb1e" (UID: "9f6836a2-3bca-4deb-a368-2da68208bb1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:32.817444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.817398 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:32.852439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.846690 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:32.915543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.915505 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\") pod \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " Oct 31 03:08:32.915714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.915609 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\") pod \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " Oct 31 03:08:32.915714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.915647 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\" (UniqueName: \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\") pod \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\" (UID: \"0c4b62f0-d4a3-43c1-a351-bfa28deebc86\") " Oct 31 03:08:32.915979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.915948 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca" pod "0c4b62f0-d4a3-43c1-a351-bfa28deebc86" (UID: "0c4b62f0-d4a3-43c1-a351-bfa28deebc86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:32.916272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.916238 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6836a2-3bca-4deb-a368-2da68208bb1e-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:32.916272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.916269 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:32.916481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.916285 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wmd\" (UniqueName: \"kubernetes.io/projected/9f6836a2-3bca-4deb-a368-2da68208bb1e-kube-api-access-s6wmd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:32.917830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.917769 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca" pod "0c4b62f0-d4a3-43c1-a351-bfa28deebc86" (UID: "0c4b62f0-d4a3-43c1-a351-bfa28deebc86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:32.917830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:32.917821 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn" pod "0c4b62f0-d4a3-43c1-a351-bfa28deebc86" (UID: "0c4b62f0-d4a3-43c1-a351-bfa28deebc86"). InnerVolumeSpecName "kube-api-access-ctftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:33.017739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.017705 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:33.017739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.017737 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctftn\" (UniqueName: \"kubernetes.io/projected/0c4b62f0-d4a3-43c1-a351-bfa28deebc86-kube-api-access-ctftn\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:33.155485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.155448 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:33.155869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.155530 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:33.515375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.515339 4556 generic.go:356] "Generic (PLEG): container finished" podID="baf49352-b6c8-49f1-9ef6-a6da94185c2a" containerID="5b884206185b10153326c3c807300399f1f22693ce38e45d00b05bb32329d58a" exitCode=0 Oct 31 03:08:33.515609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.515384 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" event={"ID":"baf49352-b6c8-49f1-9ef6-a6da94185c2a","Type":"ContainerDied","Data":"5b884206185b10153326c3c807300399f1f22693ce38e45d00b05bb32329d58a"} Oct 31 03:08:33.516759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.516738 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" Oct 31 03:08:33.516895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.516766 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6" event={"ID":"9f6836a2-3bca-4deb-a368-2da68208bb1e","Type":"ContainerDied","Data":"779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f"} Oct 31 03:08:33.516895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.516815 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779b887a53874351a5a75bc8a8399b82afb46756043aaec7b8f6dfd01f16846f" Oct 31 03:08:33.518214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.518189 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" event={"ID":"0c4b62f0-d4a3-43c1-a351-bfa28deebc86","Type":"ContainerDied","Data":"61bd00937daf19cccbeafbfae6c3e0d46ba0885e1b336243b93b253dcec56afc"} Oct 31 03:08:33.518303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.518224 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v" Oct 31 03:08:33.518303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.518234 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61bd00937daf19cccbeafbfae6c3e0d46ba0885e1b336243b93b253dcec56afc" Oct 31 03:08:33.582462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.581912 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:33.652904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:33.652871 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.124478 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf"] Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125264 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9f6836a2-3bca-4deb-a368-2da68208bb1e" containerName="revoke-keys" Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125292 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6836a2-3bca-4deb-a368-2da68208bb1e" containerName="revoke-keys" Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125352 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0c4b62f0-d4a3-43c1-a351-bfa28deebc86" containerName="revoke-keys" Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125361 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4b62f0-d4a3-43c1-a351-bfa28deebc86" containerName="revoke-keys" Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125514 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6836a2-3bca-4deb-a368-2da68208bb1e" containerName="revoke-keys" Oct 31 03:08:34.129467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.125560 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4b62f0-d4a3-43c1-a351-bfa28deebc86" containerName="revoke-keys" Oct 31 03:08:34.159055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.157833 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:34.159055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.157892 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:34.160036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.159991 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf"] Oct 31 03:08:34.160347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.160277 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.351710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.351677 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv"] Oct 31 03:08:34.374062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.366807 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.374062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.367585 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv"] Oct 31 03:08:34.446367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446337 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.446553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446515 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tt9\" (UniqueName: \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.446692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446674 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.446798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446783 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.446866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446836 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.446919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.446864 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vxth\" (UniqueName: \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.504862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.504824 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.548663 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.548725 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.548752 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxth\" (UniqueName: \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.548811 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.548905 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tt9\" (UniqueName: \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.549488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.549010 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.550715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.550571 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.550923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.550886 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.553737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.553689 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.555049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.555007 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.564826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.564764 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxth\" (UniqueName: \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\") pod \"maas-api-revoke-sub-4c4a02f7-gkbhf\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.566449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.566383 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tt9\" (UniqueName: \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\") pod \"maas-api-revoke-sub-e4523c56-w84wv\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.740507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.740472 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:34.843194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.840495 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:34.958313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:34.958270 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:35.087260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.086793 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv"] Oct 31 03:08:35.090287 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:35.090244 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743d216f_e4b8_4e66_9cab_ff6ed5378920.slice/crio-9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1 WatchSource:0}: Error finding container 9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1: Status 404 returned error can't find the container with id 9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1 Oct 31 03:08:35.118501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.118468 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\") pod \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " Oct 31 03:08:35.118763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.118739 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\") pod \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " Oct 31 03:08:35.119145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.119115 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca" pod "baf49352-b6c8-49f1-9ef6-a6da94185c2a" (UID: "baf49352-b6c8-49f1-9ef6-a6da94185c2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:35.119242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.119224 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\" (UniqueName: \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\") pod \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\" (UID: \"baf49352-b6c8-49f1-9ef6-a6da94185c2a\") " Oct 31 03:08:35.119874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.119852 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.121276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.121241 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca" pod "baf49352-b6c8-49f1-9ef6-a6da94185c2a" (UID: "baf49352-b6c8-49f1-9ef6-a6da94185c2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:35.122723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.122695 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2" pod "baf49352-b6c8-49f1-9ef6-a6da94185c2a" (UID: "baf49352-b6c8-49f1-9ef6-a6da94185c2a"). InnerVolumeSpecName "kube-api-access-gzrs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:35.155706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.155673 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:08:35.155860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.155730 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:08:35.223856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.223754 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrs2\" (UniqueName: \"kubernetes.io/projected/baf49352-b6c8-49f1-9ef6-a6da94185c2a-kube-api-access-gzrs2\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.223856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.223789 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baf49352-b6c8-49f1-9ef6-a6da94185c2a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:35.283673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.283203 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf"] Oct 31 03:08:35.341853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.341756 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:35.533289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.533256 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" event={"ID":"baf49352-b6c8-49f1-9ef6-a6da94185c2a","Type":"ContainerDied","Data":"aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a"} Oct 31 03:08:35.533531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.533312 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf83dc9d8e7ac2464967f580e704551be12710cb625905fe2d34c2577c7ad6a" Oct 31 03:08:35.533531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.533380 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx" Oct 31 03:08:35.538676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.538636 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" event={"ID":"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb","Type":"ContainerStarted","Data":"493ddaad61a15fa4259a232c962713bbca288d30b9616cea0211744970a1e33a"} Oct 31 03:08:35.540401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.540373 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" event={"ID":"743d216f-e4b8-4e66-9cab-ff6ed5378920","Type":"ContainerStarted","Data":"b5d2c17bcde29d0c4c6c91d67f05d906975236ffbe05e6c18c757821d302910a"} Oct 31 03:08:35.540532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.540448 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" event={"ID":"743d216f-e4b8-4e66-9cab-ff6ed5378920","Type":"ContainerStarted","Data":"9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1"} Oct 31 03:08:35.559157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:35.559109 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" podStartSLOduration=1.559093367 podStartE2EDuration="1.559093367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:35.557543158 +0000 UTC m=+793.412560299" watchObservedRunningTime="2026-10-31 03:08:35.559093367 +0000 UTC m=+793.414110509" Oct 31 03:08:36.132597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.132497 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:36.227321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.227287 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:36.227832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.227785 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:36.228898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.228875 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:08:36.548243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.548206 4556 generic.go:356] "Generic (PLEG): container finished" podID="9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" containerID="481df0f950b81d265e784b81b2b429e0c0511809a360d14a5454e69eea77b4f2" exitCode=0 Oct 31 03:08:36.548426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.548295 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" event={"ID":"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb","Type":"ContainerDied","Data":"481df0f950b81d265e784b81b2b429e0c0511809a360d14a5454e69eea77b4f2"} Oct 31 03:08:36.549863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.549836 4556 generic.go:356] "Generic (PLEG): container finished" podID="743d216f-e4b8-4e66-9cab-ff6ed5378920" containerID="b5d2c17bcde29d0c4c6c91d67f05d906975236ffbe05e6c18c757821d302910a" exitCode=0 Oct 31 03:08:36.549987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:36.549918 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" event={"ID":"743d216f-e4b8-4e66-9cab-ff6ed5378920","Type":"ContainerDied","Data":"b5d2c17bcde29d0c4c6c91d67f05d906975236ffbe05e6c18c757821d302910a"} Oct 31 03:08:37.010120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.010083 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:37.725767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.725711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:37.746768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.730438 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:37.783004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.782973 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:37.868158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868082 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\") pod \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " Oct 31 03:08:37.868158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868124 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\") pod \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " Oct 31 03:08:37.868409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868174 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\") pod \"743d216f-e4b8-4e66-9cab-ff6ed5378920\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " Oct 31 03:08:37.868409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868247 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\" (UniqueName: \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\") pod \"743d216f-e4b8-4e66-9cab-ff6ed5378920\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " Oct 31 03:08:37.868409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868336 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\") pod \"743d216f-e4b8-4e66-9cab-ff6ed5378920\" (UID: \"743d216f-e4b8-4e66-9cab-ff6ed5378920\") " Oct 31 03:08:37.868409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868372 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\" (UniqueName: \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\") pod \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\" (UID: \"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb\") " Oct 31 03:08:37.868656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868629 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca" pod "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" (UID: "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:37.868818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868791 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca" pod "743d216f-e4b8-4e66-9cab-ff6ed5378920" (UID: "743d216f-e4b8-4e66-9cab-ff6ed5378920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:37.868931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868911 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:37.868995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.868936 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:37.870536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.870513 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca" pod "743d216f-e4b8-4e66-9cab-ff6ed5378920" (UID: "743d216f-e4b8-4e66-9cab-ff6ed5378920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:37.870666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.870642 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca" pod "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" (UID: "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:37.870903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.870886 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth" pod "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" (UID: "9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb"). InnerVolumeSpecName "kube-api-access-5vxth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:37.870964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.870911 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9" pod "743d216f-e4b8-4e66-9cab-ff6ed5378920" (UID: "743d216f-e4b8-4e66-9cab-ff6ed5378920"). InnerVolumeSpecName "kube-api-access-l7tt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:37.970124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.970081 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tt9\" (UniqueName: \"kubernetes.io/projected/743d216f-e4b8-4e66-9cab-ff6ed5378920-kube-api-access-l7tt9\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:37.970124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.970123 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vxth\" (UniqueName: \"kubernetes.io/projected/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-kube-api-access-5vxth\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:37.970318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.970141 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:37.970318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:37.970156 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743d216f-e4b8-4e66-9cab-ff6ed5378920-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:38.438687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.438648 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:38.563711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.563624 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" Oct 31 03:08:38.563859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.563624 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf" event={"ID":"9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb","Type":"ContainerDied","Data":"493ddaad61a15fa4259a232c962713bbca288d30b9616cea0211744970a1e33a"} Oct 31 03:08:38.563859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.563776 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493ddaad61a15fa4259a232c962713bbca288d30b9616cea0211744970a1e33a" Oct 31 03:08:38.564936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.564913 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" event={"ID":"743d216f-e4b8-4e66-9cab-ff6ed5378920","Type":"ContainerDied","Data":"9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1"} Oct 31 03:08:38.565043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.564946 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9204203072e39387bcd81504ec044074baf8d20e4b88d7bf5566ac2f52e7d3a1" Oct 31 03:08:38.565043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:38.564921 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv" Oct 31 03:08:39.207180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:39.207134 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:39.933932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:39.933894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:40.534315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:40.534281 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:40.785301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:40.785190 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:41.644493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.644459 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:41.861922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.861890 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df"] Oct 31 03:08:41.862508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862491 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" containerName="revoke-keys" Oct 31 03:08:41.862583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862516 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" containerName="revoke-keys" Oct 31 03:08:41.862583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862525 4556 cpu_manager.go:414] "containerMap: removing container" podUID="baf49352-b6c8-49f1-9ef6-a6da94185c2a" containerName="revoke-keys" Oct 31 03:08:41.862583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862530 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf49352-b6c8-49f1-9ef6-a6da94185c2a" containerName="revoke-keys" Oct 31 03:08:41.862583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862541 4556 cpu_manager.go:414] "containerMap: removing container" podUID="743d216f-e4b8-4e66-9cab-ff6ed5378920" containerName="revoke-keys" Oct 31 03:08:41.862583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862546 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743d216f-e4b8-4e66-9cab-ff6ed5378920" containerName="revoke-keys" Oct 31 03:08:41.862739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862629 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf49352-b6c8-49f1-9ef6-a6da94185c2a" containerName="revoke-keys" Oct 31 03:08:41.862739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862636 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" containerName="revoke-keys" Oct 31 03:08:41.862739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.862643 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="743d216f-e4b8-4e66-9cab-ff6ed5378920" containerName="revoke-keys" Oct 31 03:08:41.866460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.866443 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:41.871852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.871777 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:41.871852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.871791 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:41.875810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.875785 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df"] Oct 31 03:08:41.929056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.929017 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:41.929230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.929186 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcmk\" (UniqueName: \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:41.929230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:41.929211 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.030592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.030558 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.030756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.030715 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcmk\" (UniqueName: \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.030756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.030747 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.031452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.031404 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.033006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.032978 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.042058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.042021 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcmk\" (UniqueName: \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\") pod \"maas-api-revoke-sub-9af8cb67-8p9df\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.182664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.182571 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:42.431040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.431002 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:42.462699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.462638 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df"] Oct 31 03:08:42.463528 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:42.463498 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda473a4dd_7bb2_46f1_9da5_ac914210c1d6.slice/crio-14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4 WatchSource:0}: Error finding container 14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4: Status 404 returned error can't find the container with id 14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4 Oct 31 03:08:42.591813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.591773 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" event={"ID":"a473a4dd-7bb2-46f1-9da5-ac914210c1d6","Type":"ContainerStarted","Data":"14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4"} Oct 31 03:08:42.927938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.927904 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:08:42.935012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.934985 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.943936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.943912 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.943967 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944029 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944068 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944086 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vr5z\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944170 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944286 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944315 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.944378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.944333 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:42.954586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:42.954556 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:08:43.045303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045222 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045322 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045346 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045363 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045390 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045485 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045524 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045735 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045779 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vr5z\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.045891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.045862 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.046188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.046164 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.046285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.046252 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.046491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.046466 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.046999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.046978 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.048656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.048631 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.048857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.048838 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.056631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.056605 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.057277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.057256 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vr5z\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.252637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.252603 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:43.432433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.432227 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:43.577474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.574709 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:08:43.615623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.604375 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:43.615623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.604513 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:43.615623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.604563 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:08:43.756610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.756563 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" event={"ID":"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42","Type":"ContainerStarted","Data":"a073d755ad06db6f89029e8a73012f3d90d8279386c1ebfb53c712d540ecd0dd"} Oct 31 03:08:43.764016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.763985 4556 generic.go:356] "Generic (PLEG): container finished" podID="a473a4dd-7bb2-46f1-9da5-ac914210c1d6" containerID="5391ad3c08eea3b7ad7f38d1cf4c826d60e4e7d4663dcfc2fef674f96e5c461e" exitCode=0 Oct 31 03:08:43.764911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:43.764710 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" event={"ID":"a473a4dd-7bb2-46f1-9da5-ac914210c1d6","Type":"ContainerDied","Data":"5391ad3c08eea3b7ad7f38d1cf4c826d60e4e7d4663dcfc2fef674f96e5c461e"} Oct 31 03:08:44.605436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.593547 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:44.806851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.806813 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" event={"ID":"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42","Type":"ContainerStarted","Data":"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838"} Oct 31 03:08:44.835019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.834967 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podStartSLOduration=2.834949899 podStartE2EDuration="2.834949899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:44.832838907 +0000 UTC m=+802.687856048" watchObservedRunningTime="2026-10-31 03:08:44.834949899 +0000 UTC m=+802.689967039" Oct 31 03:08:44.904268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.902686 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5"] Oct 31 03:08:44.918656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.918624 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5"] Oct 31 03:08:44.918814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:44.918744 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.008609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.008583 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:45.024511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.024478 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.024645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.024546 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.024645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.024634 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz29v\" (UniqueName: \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.125139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125043 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\") pod \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " Oct 31 03:08:45.125300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125283 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\") pod \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " Oct 31 03:08:45.125396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125367 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\" (UniqueName: \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\") pod \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\" (UID: \"a473a4dd-7bb2-46f1-9da5-ac914210c1d6\") " Oct 31 03:08:45.125486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125389 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca" pod "a473a4dd-7bb2-46f1-9da5-ac914210c1d6" (UID: "a473a4dd-7bb2-46f1-9da5-ac914210c1d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:45.125674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125654 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.125773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125759 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.125853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125839 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz29v\" (UniqueName: \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.126009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.125994 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:45.126497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.126473 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.127705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.127677 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk" pod "a473a4dd-7bb2-46f1-9da5-ac914210c1d6" (UID: "a473a4dd-7bb2-46f1-9da5-ac914210c1d6"). InnerVolumeSpecName "kube-api-access-tlcmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:45.127983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.127964 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca" pod "a473a4dd-7bb2-46f1-9da5-ac914210c1d6" (UID: "a473a4dd-7bb2-46f1-9da5-ac914210c1d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:45.128488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.128470 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.138224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.138196 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz29v\" (UniqueName: \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\") pod \"maas-api-revoke-sub-0e288b98-gn4b5\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.205717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.205675 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g"] Oct 31 03:08:45.206276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.206258 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a473a4dd-7bb2-46f1-9da5-ac914210c1d6" containerName="revoke-keys" Oct 31 03:08:45.206332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.206278 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a473a4dd-7bb2-46f1-9da5-ac914210c1d6" containerName="revoke-keys" Oct 31 03:08:45.206397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.206387 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a473a4dd-7bb2-46f1-9da5-ac914210c1d6" containerName="revoke-keys" Oct 31 03:08:45.210127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.210111 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.220930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.220903 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g"] Oct 31 03:08:45.227130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.227092 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:45.227130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.227133 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcmk\" (UniqueName: \"kubernetes.io/projected/a473a4dd-7bb2-46f1-9da5-ac914210c1d6-kube-api-access-tlcmk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:45.237688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.237641 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:45.252910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.252887 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:45.254311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.254285 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:45.254524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.254490 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:45.329967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.329156 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.329967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.329316 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.329967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.329498 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnz8f\" (UniqueName: \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.432264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.432232 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.432459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.432299 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnz8f\" (UniqueName: \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.432459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.432430 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.433129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.433103 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.434707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.434638 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.444122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.444096 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnz8f\" (UniqueName: \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\") pod \"maas-api-revoke-sub-31f19e94-2z87g\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.528344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.528308 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:45.551836 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.551808 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5"] Oct 31 03:08:45.554933 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:45.554886 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa56fd66_7bb4_4a66_afb5_9222fccd3294.slice/crio-bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db WatchSource:0}: Error finding container bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db: Status 404 returned error can't find the container with id bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db Oct 31 03:08:45.814356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.814318 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" event={"ID":"a473a4dd-7bb2-46f1-9da5-ac914210c1d6","Type":"ContainerDied","Data":"14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4"} Oct 31 03:08:45.814769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.814385 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f4d894f7aa32d3f6b9403f318cefe5a163a33669a36cc3617f6fb47dd3bae4" Oct 31 03:08:45.814769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.814337 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df" Oct 31 03:08:45.816046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.816019 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" event={"ID":"aa56fd66-7bb4-4a66-afb5-9222fccd3294","Type":"ContainerStarted","Data":"bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db"} Oct 31 03:08:45.836899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:45.836873 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g"] Oct 31 03:08:45.837687 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:45.837665 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c451252_2a22_44a0_823f_1e5813536fde.slice/crio-8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a WatchSource:0}: Error finding container 8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a: Status 404 returned error can't find the container with id 8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a Oct 31 03:08:45.935794 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:08:45.935760 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:08:46.253153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.253119 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:46.253313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.253173 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:46.824267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.824232 4556 generic.go:356] "Generic (PLEG): container finished" podID="aa56fd66-7bb4-4a66-afb5-9222fccd3294" containerID="af7add7329096340a632a8ca59ed6185361bdae97d3197268366593dadb3cbbf" exitCode=0 Oct 31 03:08:46.824728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.824312 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" event={"ID":"aa56fd66-7bb4-4a66-afb5-9222fccd3294","Type":"ContainerDied","Data":"af7add7329096340a632a8ca59ed6185361bdae97d3197268366593dadb3cbbf"} Oct 31 03:08:46.826116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.826092 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" event={"ID":"6c451252-2a22-44a0-823f-1e5813536fde","Type":"ContainerStarted","Data":"8bae400bd272125d7ac8e73f6cba35c9634beb36538fdbcbce6dbeef495d8c39"} Oct 31 03:08:46.826227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.826134 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" event={"ID":"6c451252-2a22-44a0-823f-1e5813536fde","Type":"ContainerStarted","Data":"8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a"} Oct 31 03:08:46.865631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:46.865577 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" podStartSLOduration=1.865561257 podStartE2EDuration="1.865561257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:46.864848419 +0000 UTC m=+804.719865561" watchObservedRunningTime="2026-10-31 03:08:46.865561257 +0000 UTC m=+804.720578400" Oct 31 03:08:47.253633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:47.253596 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:47.253819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:47.253680 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:47.874438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:47.872781 4556 generic.go:356] "Generic (PLEG): container finished" podID="6c451252-2a22-44a0-823f-1e5813536fde" containerID="8bae400bd272125d7ac8e73f6cba35c9634beb36538fdbcbce6dbeef495d8c39" exitCode=0 Oct 31 03:08:47.874438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:47.873485 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" event={"ID":"6c451252-2a22-44a0-823f-1e5813536fde","Type":"ContainerDied","Data":"8bae400bd272125d7ac8e73f6cba35c9634beb36538fdbcbce6dbeef495d8c39"} Oct 31 03:08:48.264780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.257093 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:48.264780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.257152 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:48.390459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.386504 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:48.719440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.687230 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:48.841283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.778604 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:48.841283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.783662 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:48.893437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.892513 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:08:49.002437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:48.969981 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:49.101564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.004777 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:49.120489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.019788 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:49.199805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.193345 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:08:49.291484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.272615 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:49.291484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.272684 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:49.390723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.390616 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:08:49.391378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.391356 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:08:49.435450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.434758 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:49.593440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.592432 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\" (UniqueName: \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\") pod \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " Oct 31 03:08:49.593440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.592537 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\") pod \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " Oct 31 03:08:49.593440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.592567 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\") pod \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\" (UID: \"aa56fd66-7bb4-4a66-afb5-9222fccd3294\") " Oct 31 03:08:49.618438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.599116 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v" pod "aa56fd66-7bb4-4a66-afb5-9222fccd3294" (UID: "aa56fd66-7bb4-4a66-afb5-9222fccd3294"). InnerVolumeSpecName "kube-api-access-fz29v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:49.618438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.599452 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca" pod "aa56fd66-7bb4-4a66-afb5-9222fccd3294" (UID: "aa56fd66-7bb4-4a66-afb5-9222fccd3294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:49.618438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.609478 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca" pod "aa56fd66-7bb4-4a66-afb5-9222fccd3294" (UID: "aa56fd66-7bb4-4a66-afb5-9222fccd3294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:49.694750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.694387 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:49.694750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.694451 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa56fd66-7bb4-4a66-afb5-9222fccd3294-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:49.694750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:49.694468 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz29v\" (UniqueName: \"kubernetes.io/projected/aa56fd66-7bb4-4a66-afb5-9222fccd3294-kube-api-access-fz29v\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:50.057095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.057011 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:08:50.113970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.113629 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:50.231436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.160439 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\") pod \"6c451252-2a22-44a0-823f-1e5813536fde\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " Oct 31 03:08:50.231436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.160599 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\" (UniqueName: \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\") pod \"6c451252-2a22-44a0-823f-1e5813536fde\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " Oct 31 03:08:50.231436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.160676 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\") pod \"6c451252-2a22-44a0-823f-1e5813536fde\" (UID: \"6c451252-2a22-44a0-823f-1e5813536fde\") " Oct 31 03:08:50.231436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.199264 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca" pod "6c451252-2a22-44a0-823f-1e5813536fde" (UID: "6c451252-2a22-44a0-823f-1e5813536fde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:50.259458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.255957 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:50.259458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.256029 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:50.265437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.262276 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:50.271592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.271544 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca" pod "6c451252-2a22-44a0-823f-1e5813536fde" (UID: "6c451252-2a22-44a0-823f-1e5813536fde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:50.281290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.281262 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" Oct 31 03:08:50.384650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.379015 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c451252-2a22-44a0-823f-1e5813536fde-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:50.410892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.410845 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g" event={"ID":"6c451252-2a22-44a0-823f-1e5813536fde","Type":"ContainerDied","Data":"8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a"} Oct 31 03:08:50.411140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.411120 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8002a5ea94ee806e014a6d456a390a3546d8db29cc634fd1df0168b4284a784a" Oct 31 03:08:50.411280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.411263 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" event={"ID":"aa56fd66-7bb4-4a66-afb5-9222fccd3294","Type":"ContainerDied","Data":"bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db"} Oct 31 03:08:50.411438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.411403 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9578aa6abbfdbc8e8bcdb973a696f8f1baadd0ebd75c283b27f64c176c85db" Oct 31 03:08:50.412461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.412441 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5" Oct 31 03:08:50.606655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.606616 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f" pod "6c451252-2a22-44a0-823f-1e5813536fde" (UID: "6c451252-2a22-44a0-823f-1e5813536fde"). InnerVolumeSpecName "kube-api-access-xnz8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:50.799503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.799447 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnz8f\" (UniqueName: \"kubernetes.io/projected/6c451252-2a22-44a0-823f-1e5813536fde-kube-api-access-xnz8f\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:50.972810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:50.972772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:51.256534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:51.256495 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:08:51.257174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:51.257133 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:08:52.459915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.459877 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj"] Oct 31 03:08:52.460464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460448 4556 cpu_manager.go:414] "containerMap: removing container" podUID="aa56fd66-7bb4-4a66-afb5-9222fccd3294" containerName="revoke-keys" Oct 31 03:08:52.460513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460465 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa56fd66-7bb4-4a66-afb5-9222fccd3294" containerName="revoke-keys" Oct 31 03:08:52.460513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460483 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6c451252-2a22-44a0-823f-1e5813536fde" containerName="revoke-keys" Oct 31 03:08:52.460513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460488 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c451252-2a22-44a0-823f-1e5813536fde" containerName="revoke-keys" Oct 31 03:08:52.460609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460568 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c451252-2a22-44a0-823f-1e5813536fde" containerName="revoke-keys" Oct 31 03:08:52.460609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.460582 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa56fd66-7bb4-4a66-afb5-9222fccd3294" containerName="revoke-keys" Oct 31 03:08:52.464654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.464637 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.469508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.469474 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:08:52.469687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.469661 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:08:52.474206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.474174 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj"] Oct 31 03:08:52.498217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.498180 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.498392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.498313 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.498533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.498513 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6gm\" (UniqueName: \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.600071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.600031 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6gm\" (UniqueName: \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.600268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.600150 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.600268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.600233 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.600948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.600915 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.602714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.602689 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.612240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.612212 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6gm\" (UniqueName: \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\") pod \"maas-api-revoke-sub-85e1914d-pfwmj\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:52.781849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:52.781757 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:53.031070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.031036 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:53.031694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.031666 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:53.033125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.033061 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:08:53.077487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.077459 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj"] Oct 31 03:08:53.078012 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:53.077986 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c7ad8_7b52_4fca_8715_ccb18abe117c.slice/crio-d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225 WatchSource:0}: Error finding container d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225: Status 404 returned error can't find the container with id d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225 Oct 31 03:08:53.309387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.309355 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:53.481875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:53.481841 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" event={"ID":"e70c7ad8-7b52-4fca-8715-ccb18abe117c","Type":"ContainerStarted","Data":"d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225"} Oct 31 03:08:54.489569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:54.489526 4556 generic.go:356] "Generic (PLEG): container finished" podID="e70c7ad8-7b52-4fca-8715-ccb18abe117c" containerID="d8acc0de0fe30040a46a39f86936189504aab77615da25c994c660b52fe104d7" exitCode=0 Oct 31 03:08:54.490011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:54.489618 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" event={"ID":"e70c7ad8-7b52-4fca-8715-ccb18abe117c","Type":"ContainerDied","Data":"d8acc0de0fe30040a46a39f86936189504aab77615da25c994c660b52fe104d7"} Oct 31 03:08:54.985055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:54.984998 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:55.131184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.131149 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:55.637311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.637284 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:55.748561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.748474 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\" (UniqueName: \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\") pod \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " Oct 31 03:08:55.748561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.748515 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\") pod \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " Oct 31 03:08:55.748789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.748611 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\") pod \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\" (UID: \"e70c7ad8-7b52-4fca-8715-ccb18abe117c\") " Oct 31 03:08:55.748955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.748932 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca" pod "e70c7ad8-7b52-4fca-8715-ccb18abe117c" (UID: "e70c7ad8-7b52-4fca-8715-ccb18abe117c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:55.750618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.750599 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm" pod "e70c7ad8-7b52-4fca-8715-ccb18abe117c" (UID: "e70c7ad8-7b52-4fca-8715-ccb18abe117c"). InnerVolumeSpecName "kube-api-access-dd6gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:55.750718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.750699 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca" pod "e70c7ad8-7b52-4fca-8715-ccb18abe117c" (UID: "e70c7ad8-7b52-4fca-8715-ccb18abe117c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:55.850279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.850241 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6gm\" (UniqueName: \"kubernetes.io/projected/e70c7ad8-7b52-4fca-8715-ccb18abe117c-kube-api-access-dd6gm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:55.850279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.850272 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:55.850279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:55.850282 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70c7ad8-7b52-4fca-8715-ccb18abe117c-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:56.373107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.373075 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg"] Oct 31 03:08:56.373671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.373654 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e70c7ad8-7b52-4fca-8715-ccb18abe117c" containerName="revoke-keys" Oct 31 03:08:56.373727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.373672 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70c7ad8-7b52-4fca-8715-ccb18abe117c" containerName="revoke-keys" Oct 31 03:08:56.373785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.373776 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70c7ad8-7b52-4fca-8715-ccb18abe117c" containerName="revoke-keys" Oct 31 03:08:56.377260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.377244 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.388447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.388402 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg"] Oct 31 03:08:56.456686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.456651 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.456878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.456788 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcqz\" (UniqueName: \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.456878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.456835 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.502837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.502811 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" Oct 31 03:08:56.503010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.502836 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj" event={"ID":"e70c7ad8-7b52-4fca-8715-ccb18abe117c","Type":"ContainerDied","Data":"d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225"} Oct 31 03:08:56.503010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.502881 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d396c2260e464de2ccc364c544631801070bbb6a150b3cd9439d5f4251d86225" Oct 31 03:08:56.535630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.535536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:56.558213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.558176 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.558399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.558323 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.558399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.558385 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcqz\" (UniqueName: \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.558978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.558955 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.560554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.560532 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.569715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.569687 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcqz\" (UniqueName: \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\") pod \"maas-api-revoke-sub-174f3f14-749bg\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.692922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.692892 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:56.975371 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:56.975334 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ec9799_3803_4acd_aac6_51c0f8bf1118.slice/crio-a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6 WatchSource:0}: Error finding container a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6: Status 404 returned error can't find the container with id a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6 Oct 31 03:08:56.975764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:56.975678 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg"] Oct 31 03:08:57.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:57.421893 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:57.519610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:57.519572 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" event={"ID":"d9ec9799-3803-4acd-aac6-51c0f8bf1118","Type":"ContainerStarted","Data":"e2837e04b46a80b06f6dba17d9a6690d48267c42917dd46459152de4421162af"} Oct 31 03:08:57.519782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:57.519655 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" event={"ID":"d9ec9799-3803-4acd-aac6-51c0f8bf1118","Type":"ContainerStarted","Data":"a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6"} Oct 31 03:08:57.538481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:57.538394 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" podStartSLOduration=1.538380581 podStartE2EDuration="1.538380581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:57.536000514 +0000 UTC m=+815.391017666" watchObservedRunningTime="2026-10-31 03:08:57.538380581 +0000 UTC m=+815.393397722" Oct 31 03:08:58.221317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.221287 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:58.352343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.352302 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg"] Oct 31 03:08:58.356354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.356338 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.365793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.365769 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg"] Oct 31 03:08:58.482673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.482593 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.482673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.482650 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.482851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.482831 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzrq\" (UniqueName: \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.528180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.528143 4556 generic.go:356] "Generic (PLEG): container finished" podID="d9ec9799-3803-4acd-aac6-51c0f8bf1118" containerID="e2837e04b46a80b06f6dba17d9a6690d48267c42917dd46459152de4421162af" exitCode=0 Oct 31 03:08:58.528364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.528245 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" event={"ID":"d9ec9799-3803-4acd-aac6-51c0f8bf1118","Type":"ContainerDied","Data":"e2837e04b46a80b06f6dba17d9a6690d48267c42917dd46459152de4421162af"} Oct 31 03:08:58.583639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.583603 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzrq\" (UniqueName: \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.583808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.583722 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.583808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.583765 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.584626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.584602 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.586198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.586172 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.595182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.595150 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzrq\" (UniqueName: \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\") pod \"maas-api-revoke-sub-3fc4d867-sxxgg\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.674718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.674680 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:08:58.955374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:58.955342 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg"] Oct 31 03:08:58.955766 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:08:58.955742 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec429f23_9d6e_4960_baff_fcc4d6f548b8.slice/crio-92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65 WatchSource:0}: Error finding container 92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65: Status 404 returned error can't find the container with id 92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65 Oct 31 03:08:59.031058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.031028 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:08:59.536856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.536747 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" event={"ID":"ec429f23-9d6e-4960-baff-fcc4d6f548b8","Type":"ContainerStarted","Data":"c54e3ebfaae749e9be5bea14f49c5b7841f3bb6c9d4360be7be132ea32a6f94c"} Oct 31 03:08:59.536856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.536813 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" event={"ID":"ec429f23-9d6e-4960-baff-fcc4d6f548b8","Type":"ContainerStarted","Data":"92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65"} Oct 31 03:08:59.553364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.553319 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" podStartSLOduration=1.553304084 podStartE2EDuration="1.553304084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:59.552916674 +0000 UTC m=+817.407933816" watchObservedRunningTime="2026-10-31 03:08:59.553304084 +0000 UTC m=+817.408321228" Oct 31 03:08:59.682407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.682378 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:08:59.798024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.797936 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\") pod \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " Oct 31 03:08:59.798024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.798018 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\" (UniqueName: \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\") pod \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " Oct 31 03:08:59.798205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.798157 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\") pod \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\" (UID: \"d9ec9799-3803-4acd-aac6-51c0f8bf1118\") " Oct 31 03:08:59.798544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.798510 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca" pod "d9ec9799-3803-4acd-aac6-51c0f8bf1118" (UID: "d9ec9799-3803-4acd-aac6-51c0f8bf1118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:08:59.800012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.799980 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca" pod "d9ec9799-3803-4acd-aac6-51c0f8bf1118" (UID: "d9ec9799-3803-4acd-aac6-51c0f8bf1118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:08:59.800012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.799991 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz" pod "d9ec9799-3803-4acd-aac6-51c0f8bf1118" (UID: "d9ec9799-3803-4acd-aac6-51c0f8bf1118"). InnerVolumeSpecName "kube-api-access-dhcqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:08:59.899698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.899663 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:59.899698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.899694 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ec9799-3803-4acd-aac6-51c0f8bf1118-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:59.899869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.899704 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcqz\" (UniqueName: \"kubernetes.io/projected/d9ec9799-3803-4acd-aac6-51c0f8bf1118-kube-api-access-dhcqz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:08:59.920789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:08:59.920628 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:00.545222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.545136 4556 generic.go:356] "Generic (PLEG): container finished" podID="ec429f23-9d6e-4960-baff-fcc4d6f548b8" containerID="c54e3ebfaae749e9be5bea14f49c5b7841f3bb6c9d4360be7be132ea32a6f94c" exitCode=0 Oct 31 03:09:00.545689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.545269 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" event={"ID":"ec429f23-9d6e-4960-baff-fcc4d6f548b8","Type":"ContainerDied","Data":"c54e3ebfaae749e9be5bea14f49c5b7841f3bb6c9d4360be7be132ea32a6f94c"} Oct 31 03:09:00.546727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.546706 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" event={"ID":"d9ec9799-3803-4acd-aac6-51c0f8bf1118","Type":"ContainerDied","Data":"a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6"} Oct 31 03:09:00.546727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.546724 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg" Oct 31 03:09:00.546871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.546747 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3eba455b74d41d0baa996c8e5392bbbe60e9bf8f85935413caa7d8a9d347ec6" Oct 31 03:09:00.774513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:00.774468 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:01.570159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.570124 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:01.729041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.729017 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:09:01.821751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.821668 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\") pod \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " Oct 31 03:09:01.821888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.821780 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\" (UniqueName: \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\") pod \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " Oct 31 03:09:01.821888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.821848 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\") pod \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\" (UID: \"ec429f23-9d6e-4960-baff-fcc4d6f548b8\") " Oct 31 03:09:01.822039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.822015 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca" pod "ec429f23-9d6e-4960-baff-fcc4d6f548b8" (UID: "ec429f23-9d6e-4960-baff-fcc4d6f548b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:09:01.822185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.822170 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:01.823881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.823845 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq" pod "ec429f23-9d6e-4960-baff-fcc4d6f548b8" (UID: "ec429f23-9d6e-4960-baff-fcc4d6f548b8"). InnerVolumeSpecName "kube-api-access-vwzrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:09:01.824005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.823915 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca" pod "ec429f23-9d6e-4960-baff-fcc4d6f548b8" (UID: "ec429f23-9d6e-4960-baff-fcc4d6f548b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:09:01.923174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.923134 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzrq\" (UniqueName: \"kubernetes.io/projected/ec429f23-9d6e-4960-baff-fcc4d6f548b8-kube-api-access-vwzrq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:01.923174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:01.923169 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec429f23-9d6e-4960-baff-fcc4d6f548b8-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:02.429490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.429445 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:02.560061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.559981 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" event={"ID":"ec429f23-9d6e-4960-baff-fcc4d6f548b8","Type":"ContainerDied","Data":"92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65"} Oct 31 03:09:02.560061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.560027 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92208900767a0b8b2f0384ae164075465c878f544f5b790f3f5a589c2eddde65" Oct 31 03:09:02.560061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.559991 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg" Oct 31 03:09:02.755961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.755924 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5"] Oct 31 03:09:02.756470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756454 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ec429f23-9d6e-4960-baff-fcc4d6f548b8" containerName="revoke-keys" Oct 31 03:09:02.756470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756469 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec429f23-9d6e-4960-baff-fcc4d6f548b8" containerName="revoke-keys" Oct 31 03:09:02.756574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756501 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d9ec9799-3803-4acd-aac6-51c0f8bf1118" containerName="revoke-keys" Oct 31 03:09:02.756574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756506 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ec9799-3803-4acd-aac6-51c0f8bf1118" containerName="revoke-keys" Oct 31 03:09:02.756638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756596 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ec9799-3803-4acd-aac6-51c0f8bf1118" containerName="revoke-keys" Oct 31 03:09:02.756638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.756606 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec429f23-9d6e-4960-baff-fcc4d6f548b8" containerName="revoke-keys" Oct 31 03:09:02.760145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.760129 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.764706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.764683 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:09:02.765003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.764983 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:09:02.770015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.769993 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5"] Oct 31 03:09:02.834888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.834799 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx4f5\" (UniqueName: \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.835024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.834933 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.835024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.835017 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.935634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.935599 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx4f5\" (UniqueName: \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.935784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.935688 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.935784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.935761 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.936387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.936366 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.937989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.937968 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:02.947307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:02.947282 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx4f5\" (UniqueName: \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\") pod \"maas-api-revoke-sub-81c0e3c9-tvft5\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:03.078181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.078148 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:03.237636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.237602 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:03.357017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.356983 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5"] Oct 31 03:09:03.358635 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:03.358609 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9697a4c2_87bb_4948_ab51_ea6a383a2727.slice/crio-db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748 WatchSource:0}: Error finding container db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748: Status 404 returned error can't find the container with id db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748 Oct 31 03:09:03.469162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.469122 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:09:03.570676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.570640 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" event={"ID":"9697a4c2-87bb-4948-ab51-ea6a383a2727","Type":"ContainerStarted","Data":"db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748"} Oct 31 03:09:03.768309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.768271 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:09:03.773323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.773298 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.782877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.782779 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:09:03.783946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.783921 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"distinct-w4-bebea4-kserve-self-signed-certs\"" Oct 31 03:09:03.845497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.845452 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:03.856830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.856799 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.856975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.856848 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.857021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.856996 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.857056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.857045 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlrg\" (UniqueName: \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.857205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.857187 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.857329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.857316 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958313 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958377 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958439 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958472 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlrg\" (UniqueName: \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958598 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.960486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.958708 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.964436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.962579 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.964436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.962866 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.964436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.963084 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.964436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.963733 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.968911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.968882 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:03.977862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:03.977813 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlrg\" (UniqueName: \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\") pod \"distinct-w4-bebea4-kserve-676c44d988-h8pfl\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:04.091044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.090996 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:04.342433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.338186 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:04.391462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.391398 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:09:04.394517 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:04.394488 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d01c7f_861a_47d6_a69d_935c94c338b3.slice/crio-7948398b8fe6a0f9f6424b2ea59ffa78e5a9b1a408abf21bffc7f5f8ad142cc7 WatchSource:0}: Error finding container 7948398b8fe6a0f9f6424b2ea59ffa78e5a9b1a408abf21bffc7f5f8ad142cc7: Status 404 returned error can't find the container with id 7948398b8fe6a0f9f6424b2ea59ffa78e5a9b1a408abf21bffc7f5f8ad142cc7 Oct 31 03:09:04.580165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.580075 4556 generic.go:356] "Generic (PLEG): container finished" podID="9697a4c2-87bb-4948-ab51-ea6a383a2727" containerID="51225e995d1efe04404857db0338f71a422970354f0e4289b0ea588944109cf8" exitCode=0 Oct 31 03:09:04.580329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.580156 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" event={"ID":"9697a4c2-87bb-4948-ab51-ea6a383a2727","Type":"ContainerDied","Data":"51225e995d1efe04404857db0338f71a422970354f0e4289b0ea588944109cf8"} Oct 31 03:09:04.581982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.581953 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" event={"ID":"c7d01c7f-861a-47d6-a69d-935c94c338b3","Type":"ContainerStarted","Data":"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6"} Oct 31 03:09:04.582110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.581989 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" event={"ID":"c7d01c7f-861a-47d6-a69d-935c94c338b3","Type":"ContainerStarted","Data":"7948398b8fe6a0f9f6424b2ea59ffa78e5a9b1a408abf21bffc7f5f8ad142cc7"} Oct 31 03:09:04.623449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:04.623384 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podStartSLOduration=1.623371532 podStartE2EDuration="1.623371532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:04.620500126 +0000 UTC m=+822.475517278" watchObservedRunningTime="2026-10-31 03:09:04.623371532 +0000 UTC m=+822.478388677" Oct 31 03:09:05.093431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:05.091124 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:05.782915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:05.782878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:06.101128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.101100 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:06.253866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.253771 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\" (UniqueName: \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\") pod \"9697a4c2-87bb-4948-ab51-ea6a383a2727\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " Oct 31 03:09:06.254039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.253930 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\") pod \"9697a4c2-87bb-4948-ab51-ea6a383a2727\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " Oct 31 03:09:06.254039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.253982 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\") pod \"9697a4c2-87bb-4948-ab51-ea6a383a2727\" (UID: \"9697a4c2-87bb-4948-ab51-ea6a383a2727\") " Oct 31 03:09:06.254828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.254788 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca" pod "9697a4c2-87bb-4948-ab51-ea6a383a2727" (UID: "9697a4c2-87bb-4948-ab51-ea6a383a2727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:09:06.256525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.256490 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5" pod "9697a4c2-87bb-4948-ab51-ea6a383a2727" (UID: "9697a4c2-87bb-4948-ab51-ea6a383a2727"). InnerVolumeSpecName "kube-api-access-lx4f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:09:06.258525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.258499 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca" pod "9697a4c2-87bb-4948-ab51-ea6a383a2727" (UID: "9697a4c2-87bb-4948-ab51-ea6a383a2727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:09:06.356088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.356050 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx4f5\" (UniqueName: \"kubernetes.io/projected/9697a4c2-87bb-4948-ab51-ea6a383a2727-kube-api-access-lx4f5\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:06.356088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.356085 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:06.356287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.356100 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697a4c2-87bb-4948-ab51-ea6a383a2727-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:06.608391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.608290 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" event={"ID":"9697a4c2-87bb-4948-ab51-ea6a383a2727","Type":"ContainerDied","Data":"db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748"} Oct 31 03:09:06.608391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.608327 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5" Oct 31 03:09:06.608391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.608348 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db64d572fcb07323b423dad92d23793d93c12204b53bff2c5841884862ceb748" Oct 31 03:09:06.851278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:06.851244 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:07.777425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:07.777382 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:07.940799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:07.940772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:08.698201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:08.698171 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:08.845100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:08.840042 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:08.971579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:08.971498 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:09.838973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:09.838926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:10.719319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:10.719287 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:11.143308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:11.143232 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:11.628282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:11.628249 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:12.650179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.650147 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:12.967297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.967219 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:09:12.967848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.967829 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9697a4c2-87bb-4948-ab51-ea6a383a2727" containerName="revoke-keys" Oct 31 03:09:12.967848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.967847 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9697a4c2-87bb-4948-ab51-ea6a383a2727" containerName="revoke-keys" Oct 31 03:09:12.967967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.967951 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9697a4c2-87bb-4948-ab51-ea6a383a2727" containerName="revoke-keys" Oct 31 03:09:12.971842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.971823 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:12.995971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:12.995945 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:09:13.037103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037073 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037121 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037140 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037240 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037296 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037347 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037472 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037489 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslfr\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.037524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.037508 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138582 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138735 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138758 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslfr\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138776 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138843 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138893 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.138934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138916 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.138993 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139040 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139100 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139106 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139307 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139450 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.139632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.139565 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.141285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.141257 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.141560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.141538 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.152143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.152119 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.152275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.152261 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslfr\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.289934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.289858 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:13.572966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.572939 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:09:13.573728 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:13.573702 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7abd28_dcc5_4db3_a851_caf8572900e3.slice/crio-994ca3c175b0b67a7e5783e25991bee189fe4e28f1077134046ae77aeb6c245b WatchSource:0}: Error finding container 994ca3c175b0b67a7e5783e25991bee189fe4e28f1077134046ae77aeb6c245b: Status 404 returned error can't find the container with id 994ca3c175b0b67a7e5783e25991bee189fe4e28f1077134046ae77aeb6c245b Oct 31 03:09:13.576094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.576063 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:13.576147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.576132 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:13.576219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.576162 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:13.658062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.657979 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" event={"ID":"fd7abd28-dcc5-4db3-a851-caf8572900e3","Type":"ContainerStarted","Data":"61071aa4f42af301f776e1783c2fe3589d9007e395d1eb48644fb021f3e3bfe2"} Oct 31 03:09:13.658062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.658045 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" event={"ID":"fd7abd28-dcc5-4db3-a851-caf8572900e3","Type":"ContainerStarted","Data":"994ca3c175b0b67a7e5783e25991bee189fe4e28f1077134046ae77aeb6c245b"} Oct 31 03:09:13.683646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.683582 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podStartSLOduration=1.683567713 podStartE2EDuration="1.683567713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:13.679721607 +0000 UTC m=+831.534738748" watchObservedRunningTime="2026-10-31 03:09:13.683567713 +0000 UTC m=+831.538584853" Oct 31 03:09:13.740057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:13.740018 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:14.093666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.093617 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:14.107031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.107002 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:14.292661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.292618 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:14.294346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.294319 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:14.294467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.294370 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:14.699967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.699934 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:09:14.929214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:14.929178 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:15.292971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:15.292926 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:15.293146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:15.293002 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:15.794461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:15.787804 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:15.935741 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:09:15.935704 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:09:16.293845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:16.293804 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:16.294148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:16.294120 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:16.730254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:16.730206 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:17.292846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.292807 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:17.293822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.293788 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:17.503235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.503198 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp"] Oct 31 03:09:17.508735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.508698 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.509735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.509710 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp"] Oct 31 03:09:17.515640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.515615 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:09:17.515775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.515764 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:09:17.627700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.627572 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.628022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.627788 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.628022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.627889 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.630274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.629382 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:17.729518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.729485 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.729721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.729531 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.729721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.729602 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.730171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.730147 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.731978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.731954 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.740777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.740756 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\") pod \"maas-api-revoke-sub-455945c5-9lkcp\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:17.830494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:17.830430 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:18.130440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.130398 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp"] Oct 31 03:09:18.131102 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:18.131077 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57ab690_46ae_44fe_8db3_3cccc2b26584.slice/crio-354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3 WatchSource:0}: Error finding container 354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3: Status 404 returned error can't find the container with id 354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3 Oct 31 03:09:18.299694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.299649 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:18.300179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.299719 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:18.394102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.394063 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:18.523628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.523590 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:18.716808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.716768 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" event={"ID":"c57ab690-46ae-44fe-8db3-3cccc2b26584","Type":"ContainerStarted","Data":"3814d79e80fe0274e5053403bbd3808e37d71e31a37a9f4e1616536c8006c8da"} Oct 31 03:09:18.716976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.716823 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" event={"ID":"c57ab690-46ae-44fe-8db3-3cccc2b26584","Type":"ContainerStarted","Data":"354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3"} Oct 31 03:09:18.735124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:18.735072 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" podStartSLOduration=1.735059257 podStartE2EDuration="1.735059257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:18.732949562 +0000 UTC m=+836.587966715" watchObservedRunningTime="2026-10-31 03:09:18.735059257 +0000 UTC m=+836.590076448" Oct 31 03:09:19.292908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:19.292864 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:09:19.293075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:19.292935 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 03:09:19.523671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:19.523583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:19.724085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:19.724051 4556 generic.go:356] "Generic (PLEG): container finished" podID="c57ab690-46ae-44fe-8db3-3cccc2b26584" containerID="3814d79e80fe0274e5053403bbd3808e37d71e31a37a9f4e1616536c8006c8da" exitCode=0 Oct 31 03:09:19.724399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:19.724138 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" event={"ID":"c57ab690-46ae-44fe-8db3-3cccc2b26584","Type":"ContainerDied","Data":"3814d79e80fe0274e5053403bbd3808e37d71e31a37a9f4e1616536c8006c8da"} Oct 31 03:09:20.304900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.304844 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:20.558858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.558768 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:20.559498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.559097 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:20.560346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.560313 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:09:20.882925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.882902 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:20.974018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.973981 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\") pod \"c57ab690-46ae-44fe-8db3-3cccc2b26584\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " Oct 31 03:09:20.974200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.974089 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\") pod \"c57ab690-46ae-44fe-8db3-3cccc2b26584\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " Oct 31 03:09:20.974333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.974210 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\") pod \"c57ab690-46ae-44fe-8db3-3cccc2b26584\" (UID: \"c57ab690-46ae-44fe-8db3-3cccc2b26584\") " Oct 31 03:09:20.974577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.974550 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca" pod "c57ab690-46ae-44fe-8db3-3cccc2b26584" (UID: "c57ab690-46ae-44fe-8db3-3cccc2b26584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:09:20.976027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.976004 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz" pod "c57ab690-46ae-44fe-8db3-3cccc2b26584" (UID: "c57ab690-46ae-44fe-8db3-3cccc2b26584"). InnerVolumeSpecName "kube-api-access-9fskz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:09:20.976143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:20.976125 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca" pod "c57ab690-46ae-44fe-8db3-3cccc2b26584" (UID: "c57ab690-46ae-44fe-8db3-3cccc2b26584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:09:21.075924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.075883 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:21.075924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.075916 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fskz\" (UniqueName: \"kubernetes.io/projected/c57ab690-46ae-44fe-8db3-3cccc2b26584-kube-api-access-9fskz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:21.075924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.075930 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57ab690-46ae-44fe-8db3-3cccc2b26584-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:21.138006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.134250 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:21.737829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.737788 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" event={"ID":"c57ab690-46ae-44fe-8db3-3cccc2b26584","Type":"ContainerDied","Data":"354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3"} Oct 31 03:09:21.737829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.737817 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp" Oct 31 03:09:21.738276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.737851 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="354512f6a274ef152503a65a648f919d5f2f8d07540c84cff04779dd3ff683a3" Oct 31 03:09:21.944151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:21.944110 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:22.845686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:22.845644 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:23.744426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:23.744377 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:24.940438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:24.939323 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:24.978441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:24.977925 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:09:25.022149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.022096 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c57ab690-46ae-44fe-8db3-3cccc2b26584" containerName="revoke-keys" Oct 31 03:09:25.099453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.096459 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57ab690-46ae-44fe-8db3-3cccc2b26584" containerName="revoke-keys" Oct 31 03:09:25.103435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.101777 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57ab690-46ae-44fe-8db3-3cccc2b26584" containerName="revoke-keys" Oct 31 03:09:25.145006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.144972 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:09:25.145362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.145307 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.177006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.174978 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"distinct-w3-99bc97-kserve-self-signed-certs\"" Oct 31 03:09:25.253458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.253402 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.253786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.253758 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfnkb\" (UniqueName: \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.253935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.253915 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.254056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.254039 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.254229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.254212 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.254375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.254356 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364102 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364272 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfnkb\" (UniqueName: \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364322 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364350 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364482 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.364571 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.367436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.365113 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.377440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.368188 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.377440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.368512 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.394098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.394037 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.415532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.415489 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.431564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.431535 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfnkb\" (UniqueName: \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\") pod \"distinct-w3-99bc97-kserve-7d6cf64759-525jx\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.473593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.473499 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:25.625324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.625279 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:25.921048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:25.921009 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:09:25.922349 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:25.922323 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod914c3f84_6dbe_4652_a97d_6125aa9db86d.slice/crio-fb2ddaf66dacbc5e8c06ef5c1b8c0c95672e7ab1a179989797b2eb7b2658880b WatchSource:0}: Error finding container fb2ddaf66dacbc5e8c06ef5c1b8c0c95672e7ab1a179989797b2eb7b2658880b: Status 404 returned error can't find the container with id fb2ddaf66dacbc5e8c06ef5c1b8c0c95672e7ab1a179989797b2eb7b2658880b Oct 31 03:09:26.358072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:26.357975 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:26.819841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:26.819807 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" event={"ID":"914c3f84-6dbe-4652-a97d-6125aa9db86d","Type":"ContainerStarted","Data":"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476"} Oct 31 03:09:26.820029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:26.819851 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" event={"ID":"914c3f84-6dbe-4652-a97d-6125aa9db86d","Type":"ContainerStarted","Data":"fb2ddaf66dacbc5e8c06ef5c1b8c0c95672e7ab1a179989797b2eb7b2658880b"} Oct 31 03:09:26.840077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:26.840030 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podStartSLOduration=2.84001451 podStartE2EDuration="2.84001451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:26.836632475 +0000 UTC m=+844.691649615" watchObservedRunningTime="2026-10-31 03:09:26.84001451 +0000 UTC m=+844.695031651" Oct 31 03:09:27.032288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:27.032253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:27.625947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:27.625916 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:27.915070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:27.915029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:28.084882 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.084836 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:09:28.089256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.089232 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.101019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.100990 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:09:28.205529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205487 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.205666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205605 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.205732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205668 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkjg\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.205780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205759 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.205860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.205902 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.205877 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.206018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.206004 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.206055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.206033 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.206055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.206049 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.257436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.257392 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:09:28.261623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.261606 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.279062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.279030 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"kube-root-ca.crt\"" Oct 31 03:09:28.279180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.279112 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"default-dockercfg-zjnj2\"" Oct 31 03:09:28.282867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.282843 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:09:28.284866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.284848 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"distinct-2-w6-7d8daf-kserve-self-signed-certs\"" Oct 31 03:09:28.285186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.285169 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d8daf\"/\"openshift-service-ca.crt\"" Oct 31 03:09:28.307991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.307959 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308032 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308070 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308100 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.308191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308155 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.308361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308339 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308368 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308390 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308432 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308433 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308558 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.308659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308626 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.308659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308642 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308669 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.308982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.308962 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.309051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.309027 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.309162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.309124 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.309237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.309165 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhqr\" (UniqueName: \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.309237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.309218 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkjg\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.309453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.309433 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.310845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.310825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.311298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.311279 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410248 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410327 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhqr\" (UniqueName: \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410474 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410529 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410650 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.410713 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.411227 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.411491 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.419154 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.462506 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.462512 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.467186 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.467629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.467333 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.468732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.468690 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.469245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.469194 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkjg\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.469503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.469479 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.470903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.470877 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhqr\" (UniqueName: \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\") pod \"distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.484474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.483438 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:09:28.512131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512101 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512165 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512267 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512314 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrt8\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512430 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512493 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512529 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512634 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.512734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.512687 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.573971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.573924 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:28.613990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.613951 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.613990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.613990 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614009 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrt8\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614041 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614069 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614094 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614575 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.614711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614630 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614766 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614819 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614826 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.614976 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.615067 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.615307 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.617348 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.618772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.617388 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.626237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.626209 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrt8\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.626628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.626581 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.740031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.739947 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:28.790617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.790582 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:28.867568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:28.867134 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:09:28.876544 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:28.872976 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0846676f_1acd_4a0e_9c38_3dcf0243ba74.slice/crio-d83f77cad6b3ffeec3b9599191475ff286ac50e7de984d69afcad785d405ce04 WatchSource:0}: Error finding container d83f77cad6b3ffeec3b9599191475ff286ac50e7de984d69afcad785d405ce04: Status 404 returned error can't find the container with id d83f77cad6b3ffeec3b9599191475ff286ac50e7de984d69afcad785d405ce04 Oct 31 03:09:29.028675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.028584 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:09:29.039501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.039467 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.039584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.039536 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.039584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.039564 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.089311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.089285 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:09:29.089838 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:29.089812 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dbc58d_dba2_4c13_92d5_00b08f9c78ab.slice/crio-23e8a96c60b443b8cd4c1e91f8339982335123978b4b80faafcd7026ef58e685 WatchSource:0}: Error finding container 23e8a96c60b443b8cd4c1e91f8339982335123978b4b80faafcd7026ef58e685: Status 404 returned error can't find the container with id 23e8a96c60b443b8cd4c1e91f8339982335123978b4b80faafcd7026ef58e685 Oct 31 03:09:29.092458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.092427 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.092545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.092492 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.092545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.092526 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:29.132831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.132793 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:29.851839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.851805 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" event={"ID":"ac189756-7a86-45f8-aa2d-7f0f7399f4e3","Type":"ContainerStarted","Data":"a4e3a13d4e734696c9e35fb38d567d83a36b3dfd6247415522255438cb12dbb0"} Oct 31 03:09:29.852227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.851868 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" event={"ID":"ac189756-7a86-45f8-aa2d-7f0f7399f4e3","Type":"ContainerStarted","Data":"a77a3387fed66b45b221e85213877c295d3dfd1d78ee8deca06a196456388d15"} Oct 31 03:09:29.853532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.853504 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" event={"ID":"0846676f-1acd-4a0e-9c38-3dcf0243ba74","Type":"ContainerStarted","Data":"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f"} Oct 31 03:09:29.853662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.853545 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" event={"ID":"0846676f-1acd-4a0e-9c38-3dcf0243ba74","Type":"ContainerStarted","Data":"d83f77cad6b3ffeec3b9599191475ff286ac50e7de984d69afcad785d405ce04"} Oct 31 03:09:29.855600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.855471 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" event={"ID":"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab","Type":"ContainerStarted","Data":"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345"} Oct 31 03:09:29.855600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.855518 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" event={"ID":"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab","Type":"ContainerStarted","Data":"23e8a96c60b443b8cd4c1e91f8339982335123978b4b80faafcd7026ef58e685"} Oct 31 03:09:29.899438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.896720 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podStartSLOduration=1.8967000889999999 podStartE2EDuration="1.896700089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.888208427 +0000 UTC m=+847.743225570" watchObservedRunningTime="2026-10-31 03:09:29.896700089 +0000 UTC m=+847.751717231" Oct 31 03:09:29.923655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.923590 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podStartSLOduration=1.9235707930000001 podStartE2EDuration="1.923570793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.912888325 +0000 UTC m=+847.767905486" watchObservedRunningTime="2026-10-31 03:09:29.923570793 +0000 UTC m=+847.778587934" Oct 31 03:09:29.934246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:29.934198 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podStartSLOduration=1.934185357 podStartE2EDuration="1.934185357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:29.931870093 +0000 UTC m=+847.786887235" watchObservedRunningTime="2026-10-31 03:09:29.934185357 +0000 UTC m=+847.789202889" Oct 31 03:09:30.104510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.101916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:30.576441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.574509 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:30.740862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.740833 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:30.741838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.741813 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.741917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.741871 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 03:09:30.797442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.795894 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:30.797442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.795957 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:30.797442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.796215 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:30.811749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:30.805771 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:31.232999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:31.232969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:31.741089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:31.741061 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.741321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:31.741297 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 03:09:31.800588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:31.800555 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:31.800863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:31.800837 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:32.392958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:32.392917 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:32.740804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:32.740671 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.740804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:32.740724 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 03:09:32.802438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:32.801949 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:32.802438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:32.802023 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:33.638454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:33.638399 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:33.745429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:33.743398 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.745429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:33.743471 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 03:09:33.792440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:33.791833 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:33.792440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:33.791895 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:34.740325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:34.740247 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.740325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:34.740303 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 03:09:34.795438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:34.795224 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:34.795438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:34.795282 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:34.804863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:34.802334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:35.625941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.625903 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:35.642883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.642856 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:35.752060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.752013 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:35.793886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.793853 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 03:09:35.794184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.794141 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 03:09:35.928445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.927160 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:35.928716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.928692 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:35.930200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.930175 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:09:35.968510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:35.968472 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:09:36.639549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:36.639516 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:36.795289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:36.795261 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:36.795821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:36.795786 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:36.796696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:36.796675 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:09:37.414594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.414561 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:09:37.420006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.419982 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.427110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.427083 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebea4\"/\"distinct-2-w4-bebea4-kserve-self-signed-certs\"" Oct 31 03:09:37.429012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.428982 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:09:37.486717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.486685 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:37.594324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594278 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.594543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594343 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.594543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594377 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.594543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594472 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4b6\" (UniqueName: \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.594730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594706 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.594791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.594765 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.621270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.621234 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:37.696541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696446 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696517 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696544 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696576 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696602 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696651 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4b6\" (UniqueName: \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696874 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.696977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.696888 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.697215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.697188 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.698901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.698873 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.699158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.699137 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.707511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.707489 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4b6\" (UniqueName: \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\") pod \"distinct-2-w4-bebea4-kserve-c95455c75-jrn5r\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:37.742198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:37.742173 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:38.026835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.026802 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:09:38.027298 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:38.027261 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2269b73b_86ae_4697_be99_cffa142e11ef.slice/crio-7538f35f6027def0cc947de02ef1584e8823ab3fc839c8fd8540d60dde481da0 WatchSource:0}: Error finding container 7538f35f6027def0cc947de02ef1584e8823ab3fc839c8fd8540d60dde481da0: Status 404 returned error can't find the container with id 7538f35f6027def0cc947de02ef1584e8823ab3fc839c8fd8540d60dde481da0 Oct 31 03:09:38.421486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.421379 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:38.575123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.575089 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:38.587291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.587265 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:38.977870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.977830 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" event={"ID":"2269b73b-86ae-4697-be99-cffa142e11ef","Type":"ContainerStarted","Data":"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112"} Oct 31 03:09:38.978019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.977880 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" event={"ID":"2269b73b-86ae-4697-be99-cffa142e11ef","Type":"ContainerStarted","Data":"7538f35f6027def0cc947de02ef1584e8823ab3fc839c8fd8540d60dde481da0"} Oct 31 03:09:38.989659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.989632 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:09:38.997593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:38.997548 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podStartSLOduration=1.997534778 podStartE2EDuration="1.997534778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:38.996319613 +0000 UTC m=+856.851336755" watchObservedRunningTime="2026-10-31 03:09:38.997534778 +0000 UTC m=+856.852551918" Oct 31 03:09:39.214896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:39.214853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:39.743297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:39.743266 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:39.924116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:39.924078 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:40.810690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:40.810649 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:41.193941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:41.193895 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:41.713771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:41.713733 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:42.173058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.173024 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:09:42.526698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.526604 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:42.780182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.780090 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:09:42.785130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.785105 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.798971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.798942 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:09:42.894707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.894661 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.895163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.895134 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.895520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.895377 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8m9\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.895901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.895866 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.896097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.896080 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.896302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.896276 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.896519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.896501 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.896757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.896729 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:42.896917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:42.896902 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.001235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.001196 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8m9\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.007471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.007407 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.007707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.007690 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.007871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.007857 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.008014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.007999 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.008144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.008130 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.028533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.028495 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.028804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.028779 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.028999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.028979 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.038476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.033892 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.038476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.013388 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.038476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.034666 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.038476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.014108 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.038476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.034955 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.056883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.053110 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.058806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.057996 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.103373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.103303 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.166439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.156932 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:09:43.166439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.162857 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8m9\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.342373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.342286 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:09:43.343185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.343158 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.424438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.421937 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475318 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwzl\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475397 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475461 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475526 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475600 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475632 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475700 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475750 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.484439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.475850 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.580933 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.592631 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.592740 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.604974 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605079 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605118 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605202 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605273 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605409 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605592 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwzl\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.605676 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.620284 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.630497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.620705 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.636710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.636656 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:43.649503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.642572 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.694455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.669770 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.703457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.697367 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.704323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.704294 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:09:43.794508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.772535 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821017 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821103 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821198 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821309 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821333 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821356 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821484 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.823436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.821539 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb5qp\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.878448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.875023 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.882478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.879064 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.884225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.884198 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwzl\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\") pod \"e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:43.929858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.929353 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930097 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930300 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930489 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930681 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930797 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.930920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.930906 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.931103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.931088 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.931244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.931227 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb5qp\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.931787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.929831 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.932216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.932195 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.932631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.932608 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.945432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.932993 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.947715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.946982 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.973612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.973545 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.984020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.983954 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb5qp\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.984785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.984738 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:43.985167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:43.985124 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:44.191444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:44.189147 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:44.191444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:44.189792 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:44.790468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:44.787172 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:09:44.806498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:44.804504 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.093463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.090499 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:45.093463 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:45.090586 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b9ef32_7ac0_4758_a7a6_e9537ebd1e53.slice/crio-ddaab44e9a6cce99b08f521efe3afe82fa132537fc13372f80afccf57e31ed02 WatchSource:0}: Error finding container ddaab44e9a6cce99b08f521efe3afe82fa132537fc13372f80afccf57e31ed02: Status 404 returned error can't find the container with id ddaab44e9a6cce99b08f521efe3afe82fa132537fc13372f80afccf57e31ed02 Oct 31 03:09:45.093463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.092881 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:09:45.202612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.185445 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:45.202612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.185545 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:45.202612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.185592 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:45.524026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.523986 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" event={"ID":"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53","Type":"ContainerStarted","Data":"ddaab44e9a6cce99b08f521efe3afe82fa132537fc13372f80afccf57e31ed02"} Oct 31 03:09:45.861399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.861362 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:45.862879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.862849 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:09:45.893436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:45.891925 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:09:46.090702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.089310 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.090702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.089981 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.090702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.090346 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.162796 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:09:46.162762 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:09:46.207898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.207848 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.223818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.209357 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.223818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.221614 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:09:46.230224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.230197 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:09:46.554440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.550404 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" event={"ID":"56271477-1bf7-48ae-a53f-bcd2f33afe18","Type":"ContainerStarted","Data":"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6"} Oct 31 03:09:46.554440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.550535 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" event={"ID":"56271477-1bf7-48ae-a53f-bcd2f33afe18","Type":"ContainerStarted","Data":"89e70e92357e935b796bb22bafa010483fce14975aec98e797cfba5ec3d02744"} Oct 31 03:09:46.554440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.553967 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" event={"ID":"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53","Type":"ContainerStarted","Data":"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700"} Oct 31 03:09:46.576447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.575830 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" event={"ID":"6c17ca69-f4d4-4fa3-9e4b-6d668306b360","Type":"ContainerStarted","Data":"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7"} Oct 31 03:09:46.576447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.575917 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" event={"ID":"6c17ca69-f4d4-4fa3-9e4b-6d668306b360","Type":"ContainerStarted","Data":"2a71699192fae01bf450979a0e0f2b94f3ff8dc8a463253551fa887854600f00"} Oct 31 03:09:46.584435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.581147 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podStartSLOduration=3.58112042 podStartE2EDuration="3.58112042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.574350963 +0000 UTC m=+864.429368104" watchObservedRunningTime="2026-10-31 03:09:46.58112042 +0000 UTC m=+864.436137562" Oct 31 03:09:46.605488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.605332 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podStartSLOduration=4.605314998 podStartE2EDuration="4.605314998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.596648952 +0000 UTC m=+864.451666095" watchObservedRunningTime="2026-10-31 03:09:46.605314998 +0000 UTC m=+864.460332141" Oct 31 03:09:46.629782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.629638 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podStartSLOduration=3.629616544 podStartE2EDuration="3.629616544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:46.617629199 +0000 UTC m=+864.472646338" watchObservedRunningTime="2026-10-31 03:09:46.629616544 +0000 UTC m=+864.484633685" Oct 31 03:09:46.989337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:46.988866 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.190912 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.190992 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.195619 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.195683 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.196019 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.196438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.196054 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:47.439447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.430491 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:47.439447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.435862 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:47.439447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.435916 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:47.749698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.747857 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:47.864441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:47.862622 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:48.149169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.146065 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:48.193568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.193107 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.193568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.193169 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:48.193918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.193895 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.194048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.194028 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:48.426438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.425721 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:48.426438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.425780 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:48.661901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:48.661869 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:09:49.193440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.190980 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.193440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.191041 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:49.193440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.191468 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.193440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.191507 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:49.247446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.246946 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:49.425438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.425032 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:49.425438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:49.425089 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:50.194438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.194028 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.194438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.194090 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:50.195007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.194537 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.195007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.194573 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:50.344269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.343901 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:50.426183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.422462 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m"] Oct 31 03:09:50.430379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.430347 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:50.430625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.430593 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:50.564099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.564000 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.564497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.563883 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m"] Oct 31 03:09:50.567914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.567301 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:09:50.578538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.578509 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:09:50.654839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.654247 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d857h\" (UniqueName: \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.654839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.654309 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.654839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.654337 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.756098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.756049 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d857h\" (UniqueName: \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.756389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.756111 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.756389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.756157 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.757787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.757760 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.760480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.760456 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.842506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.841592 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d857h\" (UniqueName: \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\") pod \"maas-api-revoke-sub-00f869c9-w2c9m\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:50.965303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:50.965267 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:51.237440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.192124 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.237440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.192194 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:51.237440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.193506 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.237440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.193557 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:51.453953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.453917 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:51.457677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.455474 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:51.707636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.707583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:51.852927 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:51.852881 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2804be_3e95_4a89_9651_2e6eca65150a.slice/crio-90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7 WatchSource:0}: Error finding container 90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7: Status 404 returned error can't find the container with id 90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7 Oct 31 03:09:51.864014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:51.863951 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m"] Oct 31 03:09:52.196570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.196531 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.196917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.196887 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:52.197494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.196821 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.197658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.197636 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:52.447155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.445620 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:09:52.447155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.445739 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 03:09:52.537161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.537093 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:52.660577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.660546 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:52.870680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.870503 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" event={"ID":"4b2804be-3e95-4a89-9651-2e6eca65150a","Type":"ContainerStarted","Data":"b8b536918eb7ee40390ab8bad4612ece90f3476d70fe81593f909abd9f1bbf03"} Oct 31 03:09:52.870680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.870556 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" event={"ID":"4b2804be-3e95-4a89-9651-2e6eca65150a","Type":"ContainerStarted","Data":"90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7"} Oct 31 03:09:52.893825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:52.893751 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" podStartSLOduration=2.893731751 podStartE2EDuration="2.893731751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:52.889083368 +0000 UTC m=+870.744100512" watchObservedRunningTime="2026-10-31 03:09:52.893731751 +0000 UTC m=+870.748748894" Oct 31 03:09:53.190881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.190834 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:09:53.191067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.190914 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 03:09:53.191067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.190834 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:09:53.191067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.190996 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 03:09:53.515224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.515094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:53.701912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.701880 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:53.702558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.702533 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:53.703431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.703394 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:09:53.818331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.818236 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc"] Oct 31 03:09:53.824793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.824758 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.863598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.861892 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc"] Oct 31 03:09:53.880451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.880397 4556 generic.go:356] "Generic (PLEG): container finished" podID="4b2804be-3e95-4a89-9651-2e6eca65150a" containerID="b8b536918eb7ee40390ab8bad4612ece90f3476d70fe81593f909abd9f1bbf03" exitCode=0 Oct 31 03:09:53.880836 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.880803 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" event={"ID":"4b2804be-3e95-4a89-9651-2e6eca65150a","Type":"ContainerDied","Data":"b8b536918eb7ee40390ab8bad4612ece90f3476d70fe81593f909abd9f1bbf03"} Oct 31 03:09:53.881771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.881742 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8x5\" (UniqueName: \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.881883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.881809 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.882012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.881990 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.983560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.983525 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8x5\" (UniqueName: \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.983733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.983591 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.983826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.983742 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.984561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.984533 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:53.989155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.989124 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:54.005724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:53.998972 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8x5\" (UniqueName: \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\") pod \"maas-api-revoke-sub-51706f70-sbprc\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:54.171279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.171235 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:54.197740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.197702 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:54.198243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.198155 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:54.199756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.199697 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:09:54.304302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.304266 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:54.304845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.304695 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:54.306116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.306093 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:09:54.474529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.474502 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc"] Oct 31 03:09:54.475743 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:09:54.475718 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53de3723_ad06_4f28_992a_6dcbffeac454.slice/crio-b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77 WatchSource:0}: Error finding container b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77: Status 404 returned error can't find the container with id b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77 Oct 31 03:09:54.721280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.721249 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:54.889106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.889069 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" event={"ID":"53de3723-ad06-4f28-992a-6dcbffeac454","Type":"ContainerStarted","Data":"a9c1da6809cafa99fc2ea5ed0d6f68c4a4bb7fe952b067dec1f6063d7e6e976e"} Oct 31 03:09:54.889292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.889124 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" event={"ID":"53de3723-ad06-4f28-992a-6dcbffeac454","Type":"ContainerStarted","Data":"b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77"} Oct 31 03:09:54.929254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:54.929184 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" podStartSLOduration=1.929151887 podStartE2EDuration="1.929151887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:09:54.928714001 +0000 UTC m=+872.783731142" watchObservedRunningTime="2026-10-31 03:09:54.929151887 +0000 UTC m=+872.784169025" Oct 31 03:09:55.050771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.050742 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:55.101566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.101520 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\" (UniqueName: \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\") pod \"4b2804be-3e95-4a89-9651-2e6eca65150a\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " Oct 31 03:09:55.101566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.101576 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\") pod \"4b2804be-3e95-4a89-9651-2e6eca65150a\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " Oct 31 03:09:55.101825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.101629 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\") pod \"4b2804be-3e95-4a89-9651-2e6eca65150a\" (UID: \"4b2804be-3e95-4a89-9651-2e6eca65150a\") " Oct 31 03:09:55.103763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.103731 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h" pod "4b2804be-3e95-4a89-9651-2e6eca65150a" (UID: "4b2804be-3e95-4a89-9651-2e6eca65150a"). InnerVolumeSpecName "kube-api-access-d857h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:09:55.104779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.104748 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca" pod "4b2804be-3e95-4a89-9651-2e6eca65150a" (UID: "4b2804be-3e95-4a89-9651-2e6eca65150a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:09:55.106449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.106342 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca" pod "4b2804be-3e95-4a89-9651-2e6eca65150a" (UID: "4b2804be-3e95-4a89-9651-2e6eca65150a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:09:55.203961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.203921 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d857h\" (UniqueName: \"kubernetes.io/projected/4b2804be-3e95-4a89-9651-2e6eca65150a-kube-api-access-d857h\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:55.203961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.203955 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:55.203961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.203965 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b2804be-3e95-4a89-9651-2e6eca65150a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:55.546855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.546760 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:55.898321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.898224 4556 generic.go:356] "Generic (PLEG): container finished" podID="53de3723-ad06-4f28-992a-6dcbffeac454" containerID="a9c1da6809cafa99fc2ea5ed0d6f68c4a4bb7fe952b067dec1f6063d7e6e976e" exitCode=0 Oct 31 03:09:55.898321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.898307 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" event={"ID":"53de3723-ad06-4f28-992a-6dcbffeac454","Type":"ContainerDied","Data":"a9c1da6809cafa99fc2ea5ed0d6f68c4a4bb7fe952b067dec1f6063d7e6e976e"} Oct 31 03:09:55.900082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.900045 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" event={"ID":"4b2804be-3e95-4a89-9651-2e6eca65150a","Type":"ContainerDied","Data":"90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7"} Oct 31 03:09:55.900216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.900102 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b88a651d975ac971a1a933ae0f3386898c61202c057775b8bddb43d38f5bf7" Oct 31 03:09:55.900216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:55.900136 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m" Oct 31 03:09:56.439137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:56.439085 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:57.072214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.072185 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:57.142183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.142149 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\" (UniqueName: \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\") pod \"53de3723-ad06-4f28-992a-6dcbffeac454\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " Oct 31 03:09:57.142333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.142278 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\") pod \"53de3723-ad06-4f28-992a-6dcbffeac454\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " Oct 31 03:09:57.142405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.142396 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\") pod \"53de3723-ad06-4f28-992a-6dcbffeac454\" (UID: \"53de3723-ad06-4f28-992a-6dcbffeac454\") " Oct 31 03:09:57.142700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.142680 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca" pod "53de3723-ad06-4f28-992a-6dcbffeac454" (UID: "53de3723-ad06-4f28-992a-6dcbffeac454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:09:57.143024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.142999 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:57.144329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.144304 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca" pod "53de3723-ad06-4f28-992a-6dcbffeac454" (UID: "53de3723-ad06-4f28-992a-6dcbffeac454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:09:57.144466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.144394 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5" pod "53de3723-ad06-4f28-992a-6dcbffeac454" (UID: "53de3723-ad06-4f28-992a-6dcbffeac454"). InnerVolumeSpecName "kube-api-access-9f8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:09:57.206683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.206531 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:57.244655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.244615 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8x5\" (UniqueName: \"kubernetes.io/projected/53de3723-ad06-4f28-992a-6dcbffeac454-kube-api-access-9f8x5\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:57.244655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.244657 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53de3723-ad06-4f28-992a-6dcbffeac454-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:09:57.916218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.916185 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" event={"ID":"53de3723-ad06-4f28-992a-6dcbffeac454","Type":"ContainerDied","Data":"b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77"} Oct 31 03:09:57.916218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.916209 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc" Oct 31 03:09:57.916490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:57.916237 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e378000c9e8ce1a301626319df19feffa77e5dfe0e8b458cf0120a1681bc77" Oct 31 03:09:58.007331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:58.007295 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:58.806949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:58.806897 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:09:59.516207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:09:59.516106 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:00.553519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:00.553477 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:01.791914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:01.791877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:04.127774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.127727 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:04.227909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.227872 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs"] Oct 31 03:10:04.228489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228469 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4b2804be-3e95-4a89-9651-2e6eca65150a" containerName="revoke-keys" Oct 31 03:10:04.228489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228487 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2804be-3e95-4a89-9651-2e6eca65150a" containerName="revoke-keys" Oct 31 03:10:04.228670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228499 4556 cpu_manager.go:414] "containerMap: removing container" podUID="53de3723-ad06-4f28-992a-6dcbffeac454" containerName="revoke-keys" Oct 31 03:10:04.228670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228504 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53de3723-ad06-4f28-992a-6dcbffeac454" containerName="revoke-keys" Oct 31 03:10:04.228670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228651 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2804be-3e95-4a89-9651-2e6eca65150a" containerName="revoke-keys" Oct 31 03:10:04.228670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.228660 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="53de3723-ad06-4f28-992a-6dcbffeac454" containerName="revoke-keys" Oct 31 03:10:04.233705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.233684 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.242665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.242638 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:04.242958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.242937 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:04.250159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.250121 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs"] Oct 31 03:10:04.377851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.377740 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzsp\" (UniqueName: \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.378032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.377947 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.378370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.378348 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.479132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.479098 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.479340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.479252 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.479340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.479297 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzsp\" (UniqueName: \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.479925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.479894 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.481690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.481665 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.494977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.494948 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzsp\" (UniqueName: \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\") pod \"maas-api-revoke-sub-9dfe12f1-d6jqs\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.548468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.548426 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497"] Oct 31 03:10:04.552854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.552826 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.570168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.570137 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497"] Oct 31 03:10:04.590775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.590742 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:04.686692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.683244 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67dm\" (UniqueName: \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.686692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.683480 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.686692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.683533 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.784634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.784598 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67dm\" (UniqueName: \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.784825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.784747 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.784825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.784779 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.785530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.785498 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.787390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.787366 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.796726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.796701 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67dm\" (UniqueName: \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\") pod \"maas-api-revoke-sub-89c8e562-4t497\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.871730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.871693 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:04.891656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:04.891624 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs"] Oct 31 03:10:04.891776 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:04.891753 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190ac8ad_aade_4896_a39c_4a5a6ca93091.slice/crio-e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc WatchSource:0}: Error finding container e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc: Status 404 returned error can't find the container with id e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc Oct 31 03:10:05.021689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:05.021649 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:05.051122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:05.051080 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" event={"ID":"190ac8ad-aade-4896-a39c-4a5a6ca93091","Type":"ContainerStarted","Data":"e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc"} Oct 31 03:10:05.160256 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:05.160225 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a249c8_7212_4871_b5be_84e14e714f74.slice/crio-767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17 WatchSource:0}: Error finding container 767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17: Status 404 returned error can't find the container with id 767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17 Oct 31 03:10:05.160629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:05.160526 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497"] Oct 31 03:10:05.801072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:05.801034 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:06.060233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.060128 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" event={"ID":"04a249c8-7212-4871-b5be-84e14e714f74","Type":"ContainerStarted","Data":"2abef4409dcd871a121b59ae638ecd6d4539f6dcda0ee6ede5aa3665733b942b"} Oct 31 03:10:06.060233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.060226 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" event={"ID":"04a249c8-7212-4871-b5be-84e14e714f74","Type":"ContainerStarted","Data":"767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17"} Oct 31 03:10:06.061957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.061925 4556 generic.go:356] "Generic (PLEG): container finished" podID="190ac8ad-aade-4896-a39c-4a5a6ca93091" containerID="efcfff935b6f27292f0a37a9b2953c1809d2949a772b70c3ffcde758b92afbf2" exitCode=0 Oct 31 03:10:06.062078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.062004 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" event={"ID":"190ac8ad-aade-4896-a39c-4a5a6ca93091","Type":"ContainerDied","Data":"efcfff935b6f27292f0a37a9b2953c1809d2949a772b70c3ffcde758b92afbf2"} Oct 31 03:10:06.077583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.077528 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" podStartSLOduration=2.077515048 podStartE2EDuration="2.077515048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:06.076867842 +0000 UTC m=+883.931884983" watchObservedRunningTime="2026-10-31 03:10:06.077515048 +0000 UTC m=+883.932532189" Oct 31 03:10:06.304475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.304441 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:10:06.308830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.308806 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.318800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.318778 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-99bc97\"/\"distinct-2-w3-99bc97-kserve-self-signed-certs\"" Oct 31 03:10:06.327327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.327301 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:10:06.374822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.374778 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:06.405004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.404957 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.405164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.405017 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfbz\" (UniqueName: \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.405164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.405150 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.405263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.405195 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.405322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.405303 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.405374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.405346 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.507465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507027 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.507465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507101 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfbz\" (UniqueName: \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.507465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507307 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.507465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507357 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.507465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507468 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.508015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.507518 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.508455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.508381 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.508544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.508451 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.508736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.508713 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.511020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.510991 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.511276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.511258 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.520235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.520201 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfbz\" (UniqueName: \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\") pod \"distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.621783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.621704 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:06.885941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.885845 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:06.927337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:06.927302 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:10:06.927505 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:06.927309 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59be27b7_2cc1_4119_8c9c_4964760df2a7.slice/crio-7ea85a757d4d4fb12a5db5d5caa8f8dce6d9cd133e9450ef891d069d34c5b687 WatchSource:0}: Error finding container 7ea85a757d4d4fb12a5db5d5caa8f8dce6d9cd133e9450ef891d069d34c5b687: Status 404 returned error can't find the container with id 7ea85a757d4d4fb12a5db5d5caa8f8dce6d9cd133e9450ef891d069d34c5b687 Oct 31 03:10:07.072376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.072336 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" event={"ID":"59be27b7-2cc1-4119-8c9c-4964760df2a7","Type":"ContainerStarted","Data":"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a"} Oct 31 03:10:07.072376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.072391 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" event={"ID":"59be27b7-2cc1-4119-8c9c-4964760df2a7","Type":"ContainerStarted","Data":"7ea85a757d4d4fb12a5db5d5caa8f8dce6d9cd133e9450ef891d069d34c5b687"} Oct 31 03:10:07.074135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.074107 4556 generic.go:356] "Generic (PLEG): container finished" podID="04a249c8-7212-4871-b5be-84e14e714f74" containerID="2abef4409dcd871a121b59ae638ecd6d4539f6dcda0ee6ede5aa3665733b942b" exitCode=0 Oct 31 03:10:07.074268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.074189 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" event={"ID":"04a249c8-7212-4871-b5be-84e14e714f74","Type":"ContainerDied","Data":"2abef4409dcd871a121b59ae638ecd6d4539f6dcda0ee6ede5aa3665733b942b"} Oct 31 03:10:07.093913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.093856 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podStartSLOduration=1.09384294 podStartE2EDuration="1.09384294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:07.089942568 +0000 UTC m=+884.944959712" watchObservedRunningTime="2026-10-31 03:10:07.09384294 +0000 UTC m=+884.948860081" Oct 31 03:10:07.250222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.250192 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:07.337643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.337606 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\") pod \"190ac8ad-aade-4896-a39c-4a5a6ca93091\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " Oct 31 03:10:07.338136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.337659 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\" (UniqueName: \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\") pod \"190ac8ad-aade-4896-a39c-4a5a6ca93091\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " Oct 31 03:10:07.338136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.337731 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\") pod \"190ac8ad-aade-4896-a39c-4a5a6ca93091\" (UID: \"190ac8ad-aade-4896-a39c-4a5a6ca93091\") " Oct 31 03:10:07.338136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.338073 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca" pod "190ac8ad-aade-4896-a39c-4a5a6ca93091" (UID: "190ac8ad-aade-4896-a39c-4a5a6ca93091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:07.338387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.338367 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:07.339837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.339815 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca" pod "190ac8ad-aade-4896-a39c-4a5a6ca93091" (UID: "190ac8ad-aade-4896-a39c-4a5a6ca93091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:07.339837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.339828 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp" pod "190ac8ad-aade-4896-a39c-4a5a6ca93091" (UID: "190ac8ad-aade-4896-a39c-4a5a6ca93091"). InnerVolumeSpecName "kube-api-access-jlzsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:07.439765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.439730 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzsp\" (UniqueName: \"kubernetes.io/projected/190ac8ad-aade-4896-a39c-4a5a6ca93091-kube-api-access-jlzsp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:07.439765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.439761 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/190ac8ad-aade-4896-a39c-4a5a6ca93091-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:07.622876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.622811 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:07.714558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.714522 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:07.850654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:07.850577 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:08.082484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.082451 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" Oct 31 03:10:08.082653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.082449 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs" event={"ID":"190ac8ad-aade-4896-a39c-4a5a6ca93091","Type":"ContainerDied","Data":"e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc"} Oct 31 03:10:08.082653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.082594 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9536531c6f0881dc7b68da4bdd04b961a7f2c6ff91ff78d45f2fc4ed7468fdc" Oct 31 03:10:08.251623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.251595 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:08.377478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.377426 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\") pod \"04a249c8-7212-4871-b5be-84e14e714f74\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " Oct 31 03:10:08.377962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.377509 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\") pod \"04a249c8-7212-4871-b5be-84e14e714f74\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " Oct 31 03:10:08.377962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.377657 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\" (UniqueName: \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\") pod \"04a249c8-7212-4871-b5be-84e14e714f74\" (UID: \"04a249c8-7212-4871-b5be-84e14e714f74\") " Oct 31 03:10:08.377962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.377867 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca" pod "04a249c8-7212-4871-b5be-84e14e714f74" (UID: "04a249c8-7212-4871-b5be-84e14e714f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:08.378160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.378144 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:08.379651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.379622 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca" pod "04a249c8-7212-4871-b5be-84e14e714f74" (UID: "04a249c8-7212-4871-b5be-84e14e714f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:08.379758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.379721 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm" pod "04a249c8-7212-4871-b5be-84e14e714f74" (UID: "04a249c8-7212-4871-b5be-84e14e714f74"). InnerVolumeSpecName "kube-api-access-z67dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:08.480006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.479958 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a249c8-7212-4871-b5be-84e14e714f74-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:08.480006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.480002 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67dm\" (UniqueName: \"kubernetes.io/projected/04a249c8-7212-4871-b5be-84e14e714f74-kube-api-access-z67dm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:08.500259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.500210 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:08.847644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:08.847554 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:09.092359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:09.092321 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" Oct 31 03:10:09.092359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:09.092351 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497" event={"ID":"04a249c8-7212-4871-b5be-84e14e714f74","Type":"ContainerDied","Data":"767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17"} Oct 31 03:10:09.092587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:09.092393 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767a8456926037b1dd94ceea9635eac212a2ed19eb8a5d736a0bb0878929bd17" Oct 31 03:10:09.423511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:09.421960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:10.207832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:10.207792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:11.031104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:11.031071 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:11.838883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:11.838842 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:12.609827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:12.609768 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:13.015065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015029 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:10:13.015746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015724 4556 cpu_manager.go:414] "containerMap: removing container" podUID="190ac8ad-aade-4896-a39c-4a5a6ca93091" containerName="revoke-keys" Oct 31 03:10:13.015746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015747 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190ac8ad-aade-4896-a39c-4a5a6ca93091" containerName="revoke-keys" Oct 31 03:10:13.015936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015766 4556 cpu_manager.go:414] "containerMap: removing container" podUID="04a249c8-7212-4871-b5be-84e14e714f74" containerName="revoke-keys" Oct 31 03:10:13.015936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015775 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a249c8-7212-4871-b5be-84e14e714f74" containerName="revoke-keys" Oct 31 03:10:13.016027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015961 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="190ac8ad-aade-4896-a39c-4a5a6ca93091" containerName="revoke-keys" Oct 31 03:10:13.016027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.015978 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a249c8-7212-4871-b5be-84e14e714f74" containerName="revoke-keys" Oct 31 03:10:13.019851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.019825 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.033926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.033888 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:10:13.148738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148702 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.148910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148754 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.148910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148839 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkrl\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.148910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148887 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.149013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148917 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.149013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.148937 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.149090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.149068 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.149143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.149130 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.149277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.149260 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.250991 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251073 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251165 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251279 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251333 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251372 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkrl\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251401 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251480 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251516 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251550 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251713 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.251920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251892 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.252020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.251936 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.252118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.252100 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.253742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.253720 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.253870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.253854 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.262970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.262949 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkrl\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.263079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.262974 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.340113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.340023 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:13.423369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.423300 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:13.552065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.552017 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:10:13.557862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.557840 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.573593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.573538 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:10:13.631344 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:13.631298 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b56b41_c5b0_4154_b123_b7d113bb9f6a.slice/crio-6d46576552114a471f2edca30a73115a21a6a0543223136289df17e65cdcc14c WatchSource:0}: Error finding container 6d46576552114a471f2edca30a73115a21a6a0543223136289df17e65cdcc14c: Status 404 returned error can't find the container with id 6d46576552114a471f2edca30a73115a21a6a0543223136289df17e65cdcc14c Oct 31 03:10:13.631745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.631457 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:10:13.634540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.634501 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:13.634617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.634587 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:13.634651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.634640 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:13.657661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.657630 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.657863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.657845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.657981 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658029 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658127 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658215 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658256 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658275 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.658361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.658341 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759279 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759434 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759491 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759519 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759559 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759603 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759651 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759680 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759710 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759791 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.759997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.759961 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.760508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.760119 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.760508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.760181 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.760508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.760305 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.762062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.762031 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.762885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.762857 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.771728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.771706 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.771835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.771809 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\") pod \"e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:13.880898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:13.880803 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:14.153855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.147326 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" event={"ID":"a4b56b41-c5b0-4154-b123-b7d113bb9f6a","Type":"ContainerStarted","Data":"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150"} Oct 31 03:10:14.153855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.147403 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" event={"ID":"a4b56b41-c5b0-4154-b123-b7d113bb9f6a","Type":"ContainerStarted","Data":"6d46576552114a471f2edca30a73115a21a6a0543223136289df17e65cdcc14c"} Oct 31 03:10:14.196600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.196564 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:10:14.209437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.203344 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podStartSLOduration=2.20332277 podStartE2EDuration="2.20332277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:14.191169304 +0000 UTC m=+892.046186446" watchObservedRunningTime="2026-10-31 03:10:14.20332277 +0000 UTC m=+892.058339912" Oct 31 03:10:14.221787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.220147 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:14.221787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.220242 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:14.221787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.220292 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:14.342511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.342484 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:14.343781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.343755 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:14.343920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.343803 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:14.912441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:14.898475 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:15.156228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.156190 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" event={"ID":"d7c23684-5b7f-4163-9bdc-1b3f164fc262","Type":"ContainerStarted","Data":"201dcd9d20a180a539a64b4ae9ecb5adc9dc9016635c89adc6503630222b6f28"} Oct 31 03:10:15.156408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.156272 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" event={"ID":"d7c23684-5b7f-4163-9bdc-1b3f164fc262","Type":"ContainerStarted","Data":"5eb9af0537db25baa485cd94ca8217dec62bf758ba84cb79dd995bc3517fb4df"} Oct 31 03:10:15.193513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.186252 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podStartSLOduration=2.186233743 podStartE2EDuration="2.186233743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:15.18594071 +0000 UTC m=+893.040957851" watchObservedRunningTime="2026-10-31 03:10:15.186233743 +0000 UTC m=+893.041250885" Oct 31 03:10:15.355438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.352888 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.355438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.352948 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:15.890435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.884040 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:15.890435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.884179 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:15.890435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.885574 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:15.890435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:15.885635 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:16.042706 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:10:16.042663 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:10:16.342557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.341202 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.342557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.341262 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:16.622936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.622846 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:16.636608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.636585 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:16.884177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.884085 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:16.884484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.884452 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:16.887946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.885742 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:16.930485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.930449 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8"] Oct 31 03:10:16.935161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.935143 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:16.939220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.938920 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:16.939565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.939515 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:16.940798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:16.940616 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8"] Oct 31 03:10:17.054486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.054373 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.054945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.054847 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.054945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.054921 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4fx\" (UniqueName: \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.155922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.155884 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.156070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.155975 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.156070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.156010 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4fx\" (UniqueName: \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.156623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.156602 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.158390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.158370 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.222437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.215099 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4fx\" (UniqueName: \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\") pod \"maas-api-revoke-sub-da27fc3a-f2jp8\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.237579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.237547 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:17.255094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.255067 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:17.341774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.341357 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.341774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.341439 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:17.747712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.739343 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8"] Oct 31 03:10:17.896441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.893276 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:17.896441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.893355 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:17.902520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:17.902479 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:18.230888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:18.230844 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" event={"ID":"007eeebc-f8cf-428c-aee1-aec8757a1271","Type":"ContainerStarted","Data":"2d7b4b63a836de92397fdff98ab4a594bf2ad05defb329ae7d4b6c5e0930fd70"} Oct 31 03:10:18.341108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:18.341076 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.341268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:18.341131 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:18.899492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:18.891820 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:18.899492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:18.891883 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:19.025365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.025028 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:19.240886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.240799 4556 generic.go:356] "Generic (PLEG): container finished" podID="007eeebc-f8cf-428c-aee1-aec8757a1271" containerID="fdc6d956b660efe6ca981e9dd8387badf719e40dcc3e911271a9830994c19947" exitCode=0 Oct 31 03:10:19.241340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.241175 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" event={"ID":"007eeebc-f8cf-428c-aee1-aec8757a1271","Type":"ContainerDied","Data":"fdc6d956b660efe6ca981e9dd8387badf719e40dcc3e911271a9830994c19947"} Oct 31 03:10:19.340975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.340933 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.341162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.340994 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:19.885497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.883971 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:19.885497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:19.884037 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:20.018547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.018509 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:20.343625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.342625 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.343625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.342688 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 31 03:10:20.487022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.486989 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:20.556263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.556146 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\" (UniqueName: \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\") pod \"007eeebc-f8cf-428c-aee1-aec8757a1271\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " Oct 31 03:10:20.556870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.556499 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\") pod \"007eeebc-f8cf-428c-aee1-aec8757a1271\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " Oct 31 03:10:20.556870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.556575 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\") pod \"007eeebc-f8cf-428c-aee1-aec8757a1271\" (UID: \"007eeebc-f8cf-428c-aee1-aec8757a1271\") " Oct 31 03:10:20.557552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.557523 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca" pod "007eeebc-f8cf-428c-aee1-aec8757a1271" (UID: "007eeebc-f8cf-428c-aee1-aec8757a1271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:20.561112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.561076 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca" pod "007eeebc-f8cf-428c-aee1-aec8757a1271" (UID: "007eeebc-f8cf-428c-aee1-aec8757a1271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:20.561254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.561205 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx" pod "007eeebc-f8cf-428c-aee1-aec8757a1271" (UID: "007eeebc-f8cf-428c-aee1-aec8757a1271"). InnerVolumeSpecName "kube-api-access-6d4fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:20.659452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.659389 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:20.659452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.659457 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007eeebc-f8cf-428c-aee1-aec8757a1271-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:20.659670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.659475 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4fx\" (UniqueName: \"kubernetes.io/projected/007eeebc-f8cf-428c-aee1-aec8757a1271-kube-api-access-6d4fx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:20.881935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.881837 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 31 03:10:20.881935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:20.881914 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 31 03:10:21.206825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.206772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:21.288237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.277077 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" Oct 31 03:10:21.288237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.277119 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8" event={"ID":"007eeebc-f8cf-428c-aee1-aec8757a1271","Type":"ContainerDied","Data":"2d7b4b63a836de92397fdff98ab4a594bf2ad05defb329ae7d4b6c5e0930fd70"} Oct 31 03:10:21.288237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.277190 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7b4b63a836de92397fdff98ab4a594bf2ad05defb329ae7d4b6c5e0930fd70" Oct 31 03:10:21.356800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.356760 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:21.357293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.357194 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:21.358453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.358399 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:10:21.934089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.934052 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:21.934674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.934649 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:21.935705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:21.935685 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:10:22.328169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:22.328093 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:23.202460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:23.202422 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:23.926277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:23.926237 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:24.789536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:24.789503 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:25.615160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:25.615124 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:26.421664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.421625 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:26.723905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.723820 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:10:26.724793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.724765 4556 cpu_manager.go:414] "containerMap: removing container" podUID="007eeebc-f8cf-428c-aee1-aec8757a1271" containerName="revoke-keys" Oct 31 03:10:26.724793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.724789 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007eeebc-f8cf-428c-aee1-aec8757a1271" containerName="revoke-keys" Oct 31 03:10:26.725009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.724991 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="007eeebc-f8cf-428c-aee1-aec8757a1271" containerName="revoke-keys" Oct 31 03:10:26.730552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.730529 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.737486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.737453 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:10:26.739572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.739543 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"openshift-service-ca.crt\"" Oct 31 03:10:26.747203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.747177 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"default-dockercfg-76j64\"" Oct 31 03:10:26.747331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.747214 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"distinct-2-w0-edb9e6-kserve-self-signed-certs\"" Oct 31 03:10:26.798820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.798791 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:26.834148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.834118 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-edb9e6\"/\"kube-root-ca.crt\"" Oct 31 03:10:26.895300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.895244 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.895799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.895772 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.896262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.896156 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.896457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.896276 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.896457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.896337 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflg7\" (UniqueName: \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.896457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.896393 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998423 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998329 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflg7\" (UniqueName: \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998423 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998391 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998464 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998536 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998619 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998702 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.998947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998917 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.999105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.998967 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:26.999105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:26.999045 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:27.000829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.000805 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:27.001404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.001382 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:27.011041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.011017 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflg7\" (UniqueName: \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\") pod \"distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:27.054124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.054089 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:27.337092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.337057 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:10:27.368718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.368687 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:27.594267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.594188 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:10:27.857969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.857150 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:10:27.864531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.864494 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:27.886444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:27.884750 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:10:28.020765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.020724 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drl5\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.020935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.020860 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.020935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.020895 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.020935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.020917 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.021057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.020941 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.021057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.021001 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.021209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.021176 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.021325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.021304 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.021426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.021397 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.122953 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drl5\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123027 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123057 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123074 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123103 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123137 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123175 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123247 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123332 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123535 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123649 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123763 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123832 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.123948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.123929 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.125876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.125854 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.125996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.125855 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.143130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.143097 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.143288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.143257 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drl5\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\") pod \"e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.158431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.158381 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck"] Oct 31 03:10:28.165147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.165123 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.177629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.176876 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck"] Oct 31 03:10:28.177885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.177862 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:28.177985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.177898 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:28.186396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.186372 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:28.339348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.339307 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.339880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.339845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.340169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.340148 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989gp\" (UniqueName: \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.364945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.364229 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" event={"ID":"3b77c8f0-3fd1-44c5-a139-413e999765a9","Type":"ContainerStarted","Data":"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b"} Oct 31 03:10:28.364945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.364281 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" event={"ID":"3b77c8f0-3fd1-44c5-a139-413e999765a9","Type":"ContainerStarted","Data":"63ee89c770c11c5bd706d21f80e59572921f23b5c8918004eb0ba979e3ffa22f"} Oct 31 03:10:28.389458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.389329 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podStartSLOduration=2.389307101 podStartE2EDuration="2.389307101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:28.383882297 +0000 UTC m=+906.238899439" watchObservedRunningTime="2026-10-31 03:10:28.389307101 +0000 UTC m=+906.244324244" Oct 31 03:10:28.442109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.442066 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989gp\" (UniqueName: \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.442386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.442371 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.442560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.442543 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.444122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.444091 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.446886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.446864 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.455654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.455622 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989gp\" (UniqueName: \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\") pod \"maas-api-revoke-sub-e65c5763-f8sck\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.487144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.487101 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:28.586736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.586362 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:10:28.593829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.593796 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:28.593963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.593875 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:28.593963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.593925 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:28.635095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.635044 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:28.850076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:28.849996 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck"] Oct 31 03:10:29.059470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.057525 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:29.487435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.487352 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" event={"ID":"18bba92f-ca29-4985-be28-8d1a6fca7ae1","Type":"ContainerStarted","Data":"22d879b3e796d615da529777809731576e81792d3c83b6ad4543c5bbfa503c1e"} Oct 31 03:10:29.521440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.517815 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" event={"ID":"732eaee1-4e6b-4467-a652-659ef878f820","Type":"ContainerStarted","Data":"77aabc95556893e46b9651acf7cea857ea9cc2e283d7c1f0a10e2d3f5188ee39"} Oct 31 03:10:29.521440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.517878 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" event={"ID":"732eaee1-4e6b-4467-a652-659ef878f820","Type":"ContainerStarted","Data":"2e3062a3c97f6592f56463def8ec9f3f1ddd32fd966712bbebd52ce58df02e10"} Oct 31 03:10:29.979440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.977198 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:29.980013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:29.979589 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.061441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.060613 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:10:30.191447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.191215 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:30.191623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.191460 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:30.191623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.191540 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:30.279497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.276602 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.299439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.297304 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.667441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.658619 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.667788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.667739 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:10:30.679457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.677043 4556 generic.go:356] "Generic (PLEG): container finished" podID="18bba92f-ca29-4985-be28-8d1a6fca7ae1" containerID="79dc160d45ed293be28c55e884f1ffe41c101ceadb109c215346667f40d22e7f" exitCode=0 Oct 31 03:10:30.679457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.678616 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" event={"ID":"18bba92f-ca29-4985-be28-8d1a6fca7ae1","Type":"ContainerDied","Data":"79dc160d45ed293be28c55e884f1ffe41c101ceadb109c215346667f40d22e7f"} Oct 31 03:10:30.701341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.699704 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.719449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.715880 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podStartSLOduration=3.715860999 podStartE2EDuration="3.715860999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:29.578295938 +0000 UTC m=+907.433313080" watchObservedRunningTime="2026-10-31 03:10:30.715860999 +0000 UTC m=+908.570878141" Oct 31 03:10:30.760570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.760514 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.777783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.777730 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.825061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.825021 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:30.846564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.846516 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:30.876561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.876511 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:10:30.983718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.983583 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:10:30.984753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:30.984704 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.001943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.001899 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:10:31.110749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.110684 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.171447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.113883 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.203452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.129551 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:10:31.274675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.274584 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:10:31.377625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.169509 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:10:31.434446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.342036 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:10:31.434446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.432990 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:31.653347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.653311 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:31.689445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.689388 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:31.689703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.678728 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.756582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.756544 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:10:31.757846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.677081 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:10:31.775598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.775559 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:10:31.787484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.781919 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.787484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.786304 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:10:31.822692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.711596 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.822692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.803357 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:10:31.822692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.784405 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.822692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.804677 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:10:31.862441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.851603 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:31.862441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.851990 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:10:32.023662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.984558 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:10:32.023662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.991525 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:10:32.023662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:31.993311 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:10:32.129584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.122761 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.203461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.202509 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.305504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.300863 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:10:32.305504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.300928 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.464957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.451467 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:10:32.541502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.457594 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:10:32.565817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.565724 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:32.578446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.566022 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:32.812573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.770766 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.812573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.787724 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.812573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.809681 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:10:32.840497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.832993 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.867578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.867528 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.868217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.868180 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.868672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.868649 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:10:32.900966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.886510 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:32.993441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:32.985482 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.039516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.039460 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.152640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.152598 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.308724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.308684 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:33.308998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.308967 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:33.415149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.415052 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:10:33.565609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.565572 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.649191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.649159 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:10:33.657025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.656396 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:10:33.802700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.797146 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.816912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.816826 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:10:33.834451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.833107 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:10:33.837782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.835619 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:10:33.837782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.836267 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:10:33.841157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.841137 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:10:33.843169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.843129 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:10:33.859911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.859337 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:10:33.908307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.908278 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:10:33.909407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.909381 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:10:33.938089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.937989 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:33.947896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:33.947092 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:10:34.034964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.033020 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\") pod \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " Oct 31 03:10:34.034964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.033222 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\") pod \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " Oct 31 03:10:34.034964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.033435 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\" (UniqueName: \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\") pod \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\" (UID: \"18bba92f-ca29-4985-be28-8d1a6fca7ae1\") " Oct 31 03:10:34.035259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.035128 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca" pod "18bba92f-ca29-4985-be28-8d1a6fca7ae1" (UID: "18bba92f-ca29-4985-be28-8d1a6fca7ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:34.037701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.037674 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp" pod "18bba92f-ca29-4985-be28-8d1a6fca7ae1" (UID: "18bba92f-ca29-4985-be28-8d1a6fca7ae1"). InnerVolumeSpecName "kube-api-access-989gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:34.041194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.041155 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca" pod "18bba92f-ca29-4985-be28-8d1a6fca7ae1" (UID: "18bba92f-ca29-4985-be28-8d1a6fca7ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:34.135439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.135389 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:34.135661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.135644 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-989gp\" (UniqueName: \"kubernetes.io/projected/18bba92f-ca29-4985-be28-8d1a6fca7ae1-kube-api-access-989gp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:34.135779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.135765 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bba92f-ca29-4985-be28-8d1a6fca7ae1-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:34.211619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.208910 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:34.211619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.209003 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:34.235013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.234591 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:34.364453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.364387 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" Oct 31 03:10:34.364453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.364404 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck" event={"ID":"18bba92f-ca29-4985-be28-8d1a6fca7ae1","Type":"ContainerDied","Data":"22d879b3e796d615da529777809731576e81792d3c83b6ad4543c5bbfa503c1e"} Oct 31 03:10:34.364661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.364472 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d879b3e796d615da529777809731576e81792d3c83b6ad4543c5bbfa503c1e" Oct 31 03:10:34.906153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.906104 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc"] Oct 31 03:10:34.907079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.907056 4556 cpu_manager.go:414] "containerMap: removing container" podUID="18bba92f-ca29-4985-be28-8d1a6fca7ae1" containerName="revoke-keys" Oct 31 03:10:34.907079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.907076 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bba92f-ca29-4985-be28-8d1a6fca7ae1" containerName="revoke-keys" Oct 31 03:10:34.907256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.907210 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bba92f-ca29-4985-be28-8d1a6fca7ae1" containerName="revoke-keys" Oct 31 03:10:34.911235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.911207 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:34.920514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.920485 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc"] Oct 31 03:10:34.923036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.923013 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:34.923154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.923068 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:34.952447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.948693 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:34.952447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.948763 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:34.952447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:34.949002 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzqz\" (UniqueName: \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.050907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.050862 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.051339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.050926 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.051339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.051113 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzqz\" (UniqueName: \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.051619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.051599 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.053176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.053158 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.062890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.062860 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzqz\" (UniqueName: \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\") pod \"maas-api-revoke-sub-67ef0e5f-2xlhc\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.193498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.189819 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:35.193498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.189886 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:35.230119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.230086 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:35.357172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.357133 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l"] Oct 31 03:10:35.364068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.364031 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.404997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.404964 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l"] Oct 31 03:10:35.455953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.455875 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65hr\" (UniqueName: \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.456094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.456055 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.456094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.456087 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.515779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.515726 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:35.542117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.542085 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc"] Oct 31 03:10:35.543741 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:35.543709 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558c4870_35a0_454d_bd3f_cb598fc258f6.slice/crio-f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8 WatchSource:0}: Error finding container f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8: Status 404 returned error can't find the container with id f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8 Oct 31 03:10:35.559386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.559344 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65hr\" (UniqueName: \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.559618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.559595 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.559778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.559647 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.562044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.560904 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.588501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.587167 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.601514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.601489 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65hr\" (UniqueName: \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\") pod \"maas-api-revoke-sub-009d534b-nnl7l\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.720127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.720042 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:35.840663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.839583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:35.991377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:35.985921 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:36.105865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.105834 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l"] Oct 31 03:10:36.189907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.189866 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:36.190149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.190119 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:36.427112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.427074 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" event={"ID":"558c4870-35a0-454d-bd3f-cb598fc258f6","Type":"ContainerStarted","Data":"0cad4199ec96e430ebc991af7f518bda9b82b9bcc7500b024d324aca47e92610"} Oct 31 03:10:36.427353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.427140 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" event={"ID":"558c4870-35a0-454d-bd3f-cb598fc258f6","Type":"ContainerStarted","Data":"f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8"} Oct 31 03:10:36.429060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.429033 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" event={"ID":"e839526f-1ab3-481f-9d85-22f4a3ae9f01","Type":"ContainerStarted","Data":"9fd0b4fff588cfbd29d4e4d4b8c596896e1bfc3426b18076e52638cee705a69c"} Oct 31 03:10:36.446678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.446621 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" podStartSLOduration=2.446606137 podStartE2EDuration="2.446606137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:36.443582004 +0000 UTC m=+914.298599145" watchObservedRunningTime="2026-10-31 03:10:36.446606137 +0000 UTC m=+914.301623278" Oct 31 03:10:36.531113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:36.531069 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:37.054369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.054283 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:37.067401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.067373 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:37.191979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.191944 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:37.192557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.192515 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:37.439358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.439324 4556 generic.go:356] "Generic (PLEG): container finished" podID="558c4870-35a0-454d-bd3f-cb598fc258f6" containerID="0cad4199ec96e430ebc991af7f518bda9b82b9bcc7500b024d324aca47e92610" exitCode=0 Oct 31 03:10:37.439534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.439435 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" event={"ID":"558c4870-35a0-454d-bd3f-cb598fc258f6","Type":"ContainerDied","Data":"0cad4199ec96e430ebc991af7f518bda9b82b9bcc7500b024d324aca47e92610"} Oct 31 03:10:37.441122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.441095 4556 generic.go:356] "Generic (PLEG): container finished" podID="e839526f-1ab3-481f-9d85-22f4a3ae9f01" containerID="2d2ccc03ad3755990f9700b40ee8917d2d1578846d1227994efc0613ae2687e9" exitCode=0 Oct 31 03:10:37.441260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.441152 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" event={"ID":"e839526f-1ab3-481f-9d85-22f4a3ae9f01","Type":"ContainerDied","Data":"2d2ccc03ad3755990f9700b40ee8917d2d1578846d1227994efc0613ae2687e9"} Oct 31 03:10:37.453556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.453457 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:10:37.508406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:37.508365 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:38.187003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.186966 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 03:10:38.187177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.187024 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 03:10:38.297089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.297052 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg"] Oct 31 03:10:38.301564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.301544 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.311905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.311871 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg"] Oct 31 03:10:38.437382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.437284 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.437622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.437599 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.437695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.437651 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.512586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.512476 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:38.543868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.541524 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.543868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.541595 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.543868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.541720 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.543868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.543645 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.564049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.546442 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.564049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.558776 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\") pod \"maas-api-revoke-sub-3977ac75-xfwdg\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.638657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.633968 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:38.730086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.729952 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:38.742726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.742624 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876128 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\") pod \"558c4870-35a0-454d-bd3f-cb598fc258f6\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876205 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\" (UniqueName: \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\") pod \"558c4870-35a0-454d-bd3f-cb598fc258f6\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876332 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\" (UniqueName: \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\") pod \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876363 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\") pod \"558c4870-35a0-454d-bd3f-cb598fc258f6\" (UID: \"558c4870-35a0-454d-bd3f-cb598fc258f6\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876465 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\") pod \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.876532 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\") pod \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\" (UID: \"e839526f-1ab3-481f-9d85-22f4a3ae9f01\") " Oct 31 03:10:38.877473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.877478 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca" pod "e839526f-1ab3-481f-9d85-22f4a3ae9f01" (UID: "e839526f-1ab3-481f-9d85-22f4a3ae9f01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:38.878014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.877811 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca" pod "558c4870-35a0-454d-bd3f-cb598fc258f6" (UID: "558c4870-35a0-454d-bd3f-cb598fc258f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:38.881697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.881668 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca" pod "558c4870-35a0-454d-bd3f-cb598fc258f6" (UID: "558c4870-35a0-454d-bd3f-cb598fc258f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:38.883592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.883563 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr" pod "e839526f-1ab3-481f-9d85-22f4a3ae9f01" (UID: "e839526f-1ab3-481f-9d85-22f4a3ae9f01"). InnerVolumeSpecName "kube-api-access-c65hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:38.883922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.883897 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca" pod "e839526f-1ab3-481f-9d85-22f4a3ae9f01" (UID: "e839526f-1ab3-481f-9d85-22f4a3ae9f01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:38.885211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.885182 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz" pod "558c4870-35a0-454d-bd3f-cb598fc258f6" (UID: "558c4870-35a0-454d-bd3f-cb598fc258f6"). InnerVolumeSpecName "kube-api-access-kvzqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978281 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvzqz\" (UniqueName: \"kubernetes.io/projected/558c4870-35a0-454d-bd3f-cb598fc258f6-kube-api-access-kvzqz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978316 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65hr\" (UniqueName: \"kubernetes.io/projected/e839526f-1ab3-481f-9d85-22f4a3ae9f01-kube-api-access-c65hr\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978328 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978337 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978346 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e839526f-1ab3-481f-9d85-22f4a3ae9f01-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.978370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.978356 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558c4870-35a0-454d-bd3f-cb598fc258f6-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:38.989531 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:38.989498 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d4474a_67ca_45f3_b289_7ed33204d44d.slice/crio-276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a WatchSource:0}: Error finding container 276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a: Status 404 returned error can't find the container with id 276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a Oct 31 03:10:38.989651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:38.989628 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg"] Oct 31 03:10:39.192116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.192083 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:39.192528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.192504 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:39.193272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.193253 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:10:39.472589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.472555 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" Oct 31 03:10:39.473002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.472562 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc" event={"ID":"558c4870-35a0-454d-bd3f-cb598fc258f6","Type":"ContainerDied","Data":"f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8"} Oct 31 03:10:39.473002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.472741 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d8c3793ec8744a5c139fd426cec9ecb36cd4c322d375e709c236ad9bb7afa8" Oct 31 03:10:39.474322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.474298 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" event={"ID":"e839526f-1ab3-481f-9d85-22f4a3ae9f01","Type":"ContainerDied","Data":"9fd0b4fff588cfbd29d4e4d4b8c596896e1bfc3426b18076e52638cee705a69c"} Oct 31 03:10:39.474541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.474517 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd0b4fff588cfbd29d4e4d4b8c596896e1bfc3426b18076e52638cee705a69c" Oct 31 03:10:39.474541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.474335 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l" Oct 31 03:10:39.476495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.476471 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" event={"ID":"14d4474a-67ca-45f3-b289-7ed33204d44d","Type":"ContainerStarted","Data":"5b3f03b362ddc9be82aeea1a0db84af81ec750887ee6f7532a674dec6bee81e5"} Oct 31 03:10:39.476644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.476627 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" event={"ID":"14d4474a-67ca-45f3-b289-7ed33204d44d","Type":"ContainerStarted","Data":"276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a"} Oct 31 03:10:39.478570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.478537 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:39.495688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:39.495628 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" podStartSLOduration=1.495614108 podStartE2EDuration="1.495614108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:39.491967075 +0000 UTC m=+917.346984215" watchObservedRunningTime="2026-10-31 03:10:39.495614108 +0000 UTC m=+917.350631249" Oct 31 03:10:40.233961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:40.233905 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:40.485755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:40.485670 4556 generic.go:356] "Generic (PLEG): container finished" podID="14d4474a-67ca-45f3-b289-7ed33204d44d" containerID="5b3f03b362ddc9be82aeea1a0db84af81ec750887ee6f7532a674dec6bee81e5" exitCode=0 Oct 31 03:10:40.486112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:40.485764 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" event={"ID":"14d4474a-67ca-45f3-b289-7ed33204d44d","Type":"ContainerDied","Data":"5b3f03b362ddc9be82aeea1a0db84af81ec750887ee6f7532a674dec6bee81e5"} Oct 31 03:10:41.022865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.022810 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:41.666125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.666096 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:41.728042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.728000 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\") pod \"14d4474a-67ca-45f3-b289-7ed33204d44d\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " Oct 31 03:10:41.728232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.728132 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\") pod \"14d4474a-67ca-45f3-b289-7ed33204d44d\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " Oct 31 03:10:41.728696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.728373 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\") pod \"14d4474a-67ca-45f3-b289-7ed33204d44d\" (UID: \"14d4474a-67ca-45f3-b289-7ed33204d44d\") " Oct 31 03:10:41.728696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.728680 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca" pod "14d4474a-67ca-45f3-b289-7ed33204d44d" (UID: "14d4474a-67ca-45f3-b289-7ed33204d44d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:41.729169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.729142 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:41.730838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.730807 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca" pod "14d4474a-67ca-45f3-b289-7ed33204d44d" (UID: "14d4474a-67ca-45f3-b289-7ed33204d44d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:41.731213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.731185 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm" pod "14d4474a-67ca-45f3-b289-7ed33204d44d" (UID: "14d4474a-67ca-45f3-b289-7ed33204d44d"). InnerVolumeSpecName "kube-api-access-rzvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:41.829010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.828376 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:41.839265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.839222 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/14d4474a-67ca-45f3-b289-7ed33204d44d-kube-api-access-rzvnm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:41.839265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:41.839265 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d4474a-67ca-45f3-b289-7ed33204d44d-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:42.504930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:42.502833 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" Oct 31 03:10:42.504930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:42.502938 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg" event={"ID":"14d4474a-67ca-45f3-b289-7ed33204d44d","Type":"ContainerDied","Data":"276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a"} Oct 31 03:10:42.504930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:42.502994 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="276e9fe09b23f3e4e3eeca8296c9e817ed823a65e0118c9bd0f7df5c2d09340a" Oct 31 03:10:42.635570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:42.635483 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:43.492266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492221 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z"] Oct 31 03:10:43.492916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492897 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e839526f-1ab3-481f-9d85-22f4a3ae9f01" containerName="revoke-keys" Oct 31 03:10:43.492957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492917 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e839526f-1ab3-481f-9d85-22f4a3ae9f01" containerName="revoke-keys" Oct 31 03:10:43.492957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492940 4556 cpu_manager.go:414] "containerMap: removing container" podUID="14d4474a-67ca-45f3-b289-7ed33204d44d" containerName="revoke-keys" Oct 31 03:10:43.492957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492945 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d4474a-67ca-45f3-b289-7ed33204d44d" containerName="revoke-keys" Oct 31 03:10:43.493051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492962 4556 cpu_manager.go:414] "containerMap: removing container" podUID="558c4870-35a0-454d-bd3f-cb598fc258f6" containerName="revoke-keys" Oct 31 03:10:43.493051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.492967 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558c4870-35a0-454d-bd3f-cb598fc258f6" containerName="revoke-keys" Oct 31 03:10:43.493111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.493059 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="558c4870-35a0-454d-bd3f-cb598fc258f6" containerName="revoke-keys" Oct 31 03:10:43.493111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.493067 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d4474a-67ca-45f3-b289-7ed33204d44d" containerName="revoke-keys" Oct 31 03:10:43.493111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.493074 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e839526f-1ab3-481f-9d85-22f4a3ae9f01" containerName="revoke-keys" Oct 31 03:10:43.496967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.496944 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.500123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.500097 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:43.500321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.500284 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:43.517789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.510756 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z"] Oct 31 03:10:43.548627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.546584 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:43.566610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.566570 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m67f\" (UniqueName: \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.566610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.566619 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.566904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.566752 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.667935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.667895 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m67f\" (UniqueName: \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.668089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.667942 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.668089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.668079 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.668757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.668734 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.670491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.670468 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.679500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.679460 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m67f\" (UniqueName: \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\") pod \"maas-api-revoke-sub-116c31a2-gcb6z\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:43.823556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:43.823466 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:44.135610 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:44.135577 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d44479_d6ba_4c0c_a2dc_d5d44fa9d20a.slice/crio-77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de WatchSource:0}: Error finding container 77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de: Status 404 returned error can't find the container with id 77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de Oct 31 03:10:44.136923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:44.136892 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z"] Oct 31 03:10:44.411966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:44.411900 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:44.548549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:44.548513 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" event={"ID":"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a","Type":"ContainerStarted","Data":"302b2d11c995f2667f221b366cd43b29f76a2da508824ebd8fad2b11e0850178"} Oct 31 03:10:44.549031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:44.548571 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" event={"ID":"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a","Type":"ContainerStarted","Data":"77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de"} Oct 31 03:10:44.572514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:44.572440 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" podStartSLOduration=1.572397269 podStartE2EDuration="1.572397269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:44.5679142 +0000 UTC m=+922.422931366" watchObservedRunningTime="2026-10-31 03:10:44.572397269 +0000 UTC m=+922.427414411" Oct 31 03:10:45.348033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.347937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:45.558219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.558178 4556 generic.go:356] "Generic (PLEG): container finished" podID="55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" containerID="302b2d11c995f2667f221b366cd43b29f76a2da508824ebd8fad2b11e0850178" exitCode=0 Oct 31 03:10:45.558697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.558267 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" event={"ID":"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a","Type":"ContainerDied","Data":"302b2d11c995f2667f221b366cd43b29f76a2da508824ebd8fad2b11e0850178"} Oct 31 03:10:45.715480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.715441 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b"] Oct 31 03:10:45.719935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.719914 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.729535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.729496 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b"] Oct 31 03:10:45.800866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.800820 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49th\" (UniqueName: \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.801025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.801005 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.801167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.801152 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.903283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.903238 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.903515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.903365 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.903515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.903490 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49th\" (UniqueName: \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.904108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.904082 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.905759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.905736 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.914836 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:45.914803 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49th\" (UniqueName: \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\") pod \"maas-api-revoke-sub-b7c26c39-trc9b\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:45.941529 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:10:45.941406 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:10:46.050193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.050093 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:46.106811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.106773 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x"] Oct 31 03:10:46.113578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.113331 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.124007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.121655 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x"] Oct 31 03:10:46.210384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.210347 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.210686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.210660 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdg5\" (UniqueName: \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.210894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.210865 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.245459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.243540 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:46.313117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.313029 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.313363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.313122 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdg5\" (UniqueName: \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.313363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.313215 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.314131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.314103 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.316662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.316634 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.326503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.326400 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdg5\" (UniqueName: \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\") pod \"maas-api-revoke-sub-0f9515da-hlv2x\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.339240 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:46.339202 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295b2968_837c_4e0e_98fd_cdd5b9b44f40.slice/crio-06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab WatchSource:0}: Error finding container 06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab: Status 404 returned error can't find the container with id 06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab Oct 31 03:10:46.339775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.339655 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b"] Oct 31 03:10:46.444232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.444191 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:46.572098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.572002 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" event={"ID":"295b2968-837c-4e0e-98fd-cdd5b9b44f40","Type":"ContainerStarted","Data":"06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab"} Oct 31 03:10:46.719119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.719087 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:46.731460 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:46.731380 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b7126c_a616_472a_a4d6_aab3dd3a3944.slice/crio-56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e WatchSource:0}: Error finding container 56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e: Status 404 returned error can't find the container with id 56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e Oct 31 03:10:46.732273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.732245 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x"] Oct 31 03:10:46.826308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.826214 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\") pod \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " Oct 31 03:10:46.826531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.826306 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\" (UniqueName: \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\") pod \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " Oct 31 03:10:46.826632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.826616 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\") pod \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\" (UID: \"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a\") " Oct 31 03:10:46.826986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.826955 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca" pod "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" (UID: "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:46.827370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.827349 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:46.828480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.828456 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca" pod "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" (UID: "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:46.828559 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.828493 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f" pod "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" (UID: "55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a"). InnerVolumeSpecName "kube-api-access-6m67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:46.928787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.928751 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:46.928787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:46.928786 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m67f\" (UniqueName: \"kubernetes.io/projected/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a-kube-api-access-6m67f\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:47.134191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.134110 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:47.581424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.581369 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" event={"ID":"f9b7126c-a616-472a-a4d6-aab3dd3a3944","Type":"ContainerStarted","Data":"3ea982a36a0294343d3e70039808f2451817a061de0957b264b00a79a7fc1359"} Oct 31 03:10:47.581903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.581448 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" event={"ID":"f9b7126c-a616-472a-a4d6-aab3dd3a3944","Type":"ContainerStarted","Data":"56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e"} Oct 31 03:10:47.583326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.583299 4556 generic.go:356] "Generic (PLEG): container finished" podID="295b2968-837c-4e0e-98fd-cdd5b9b44f40" containerID="fdf46e4779dd62afe140420845542eebecc64a5a8e7f11824f61905c46b755d9" exitCode=0 Oct 31 03:10:47.583461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.583434 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" event={"ID":"295b2968-837c-4e0e-98fd-cdd5b9b44f40","Type":"ContainerDied","Data":"fdf46e4779dd62afe140420845542eebecc64a5a8e7f11824f61905c46b755d9"} Oct 31 03:10:47.584896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.584877 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" Oct 31 03:10:47.584995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.584881 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z" event={"ID":"55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a","Type":"ContainerDied","Data":"77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de"} Oct 31 03:10:47.585060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.585014 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77964daf430d339caef0f9b5b0b129b28a48180e2e3ff912e1fac7ef9fe068de" Oct 31 03:10:47.600629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:47.600577 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" podStartSLOduration=1.600560698 podStartE2EDuration="1.600560698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:47.597775523 +0000 UTC m=+925.452792666" watchObservedRunningTime="2026-10-31 03:10:47.600560698 +0000 UTC m=+925.455577839" Oct 31 03:10:48.006583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.006462 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:48.597011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.596605 4556 generic.go:356] "Generic (PLEG): container finished" podID="f9b7126c-a616-472a-a4d6-aab3dd3a3944" containerID="3ea982a36a0294343d3e70039808f2451817a061de0957b264b00a79a7fc1359" exitCode=0 Oct 31 03:10:48.597011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.596699 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" event={"ID":"f9b7126c-a616-472a-a4d6-aab3dd3a3944","Type":"ContainerDied","Data":"3ea982a36a0294343d3e70039808f2451817a061de0957b264b00a79a7fc1359"} Oct 31 03:10:48.770254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.770225 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:48.858949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.858858 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\" (UniqueName: \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\") pod \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " Oct 31 03:10:48.859106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.859014 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\") pod \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " Oct 31 03:10:48.859106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.859099 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\") pod \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\" (UID: \"295b2968-837c-4e0e-98fd-cdd5b9b44f40\") " Oct 31 03:10:48.859453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.859405 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca" pod "295b2968-837c-4e0e-98fd-cdd5b9b44f40" (UID: "295b2968-837c-4e0e-98fd-cdd5b9b44f40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:48.859841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.859820 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:48.861112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.861076 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th" pod "295b2968-837c-4e0e-98fd-cdd5b9b44f40" (UID: "295b2968-837c-4e0e-98fd-cdd5b9b44f40"). InnerVolumeSpecName "kube-api-access-r49th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:48.861197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.861180 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca" pod "295b2968-837c-4e0e-98fd-cdd5b9b44f40" (UID: "295b2968-837c-4e0e-98fd-cdd5b9b44f40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:48.900542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.900501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:48.962386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.962341 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49th\" (UniqueName: \"kubernetes.io/projected/295b2968-837c-4e0e-98fd-cdd5b9b44f40-kube-api-access-r49th\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:48.962386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:48.962387 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295b2968-837c-4e0e-98fd-cdd5b9b44f40-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:49.604929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.604901 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" Oct 31 03:10:49.605301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.604900 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b" event={"ID":"295b2968-837c-4e0e-98fd-cdd5b9b44f40","Type":"ContainerDied","Data":"06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab"} Oct 31 03:10:49.605301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.605031 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e79bfeff46282a07d092785e68722abd7a963a06da1d764e1719f7919a42ab" Oct 31 03:10:49.745989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.745958 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:49.772753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.772442 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:49.877934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.877853 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\" (UniqueName: \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\") pod \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " Oct 31 03:10:49.878071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.877993 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\") pod \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " Oct 31 03:10:49.878223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.878203 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\") pod \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\" (UID: \"f9b7126c-a616-472a-a4d6-aab3dd3a3944\") " Oct 31 03:10:49.878434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.878381 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca" pod "f9b7126c-a616-472a-a4d6-aab3dd3a3944" (UID: "f9b7126c-a616-472a-a4d6-aab3dd3a3944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:49.878827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.878805 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:49.880167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.880142 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca" pod "f9b7126c-a616-472a-a4d6-aab3dd3a3944" (UID: "f9b7126c-a616-472a-a4d6-aab3dd3a3944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:49.880263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.880225 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5" pod "f9b7126c-a616-472a-a4d6-aab3dd3a3944" (UID: "f9b7126c-a616-472a-a4d6-aab3dd3a3944"). InnerVolumeSpecName "kube-api-access-pkdg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:49.980036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.980001 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdg5\" (UniqueName: \"kubernetes.io/projected/f9b7126c-a616-472a-a4d6-aab3dd3a3944-kube-api-access-pkdg5\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:49.980036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:49.980033 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b7126c-a616-472a-a4d6-aab3dd3a3944-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:50.604514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:50.604396 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:50.631082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:50.631043 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" event={"ID":"f9b7126c-a616-472a-a4d6-aab3dd3a3944","Type":"ContainerDied","Data":"56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e"} Oct 31 03:10:50.631586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:50.631112 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56effdb1306581fd361219474b641fe40fd3408b671e6e1c63fc821bad47e99e" Oct 31 03:10:50.631586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:50.631062 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x" Oct 31 03:10:51.542579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:51.542541 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:52.320100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.320062 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:52.492335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.492293 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf"] Oct 31 03:10:52.493060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493042 4556 cpu_manager.go:414] "containerMap: removing container" podUID="295b2968-837c-4e0e-98fd-cdd5b9b44f40" containerName="revoke-keys" Oct 31 03:10:52.493114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493063 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295b2968-837c-4e0e-98fd-cdd5b9b44f40" containerName="revoke-keys" Oct 31 03:10:52.493114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493073 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f9b7126c-a616-472a-a4d6-aab3dd3a3944" containerName="revoke-keys" Oct 31 03:10:52.493114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493079 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b7126c-a616-472a-a4d6-aab3dd3a3944" containerName="revoke-keys" Oct 31 03:10:52.493114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493097 4556 cpu_manager.go:414] "containerMap: removing container" podUID="55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" containerName="revoke-keys" Oct 31 03:10:52.493114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493102 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" containerName="revoke-keys" Oct 31 03:10:52.493259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493217 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" containerName="revoke-keys" Oct 31 03:10:52.493259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493231 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="295b2968-837c-4e0e-98fd-cdd5b9b44f40" containerName="revoke-keys" Oct 31 03:10:52.493259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.493237 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b7126c-a616-472a-a4d6-aab3dd3a3944" containerName="revoke-keys" Oct 31 03:10:52.496990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.496967 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.499891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.499861 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:10:52.500296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.500266 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:10:52.513960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.507556 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf"] Oct 31 03:10:52.616700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.616611 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.616848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.616746 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fdh\" (UniqueName: \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.616996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.616973 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.718661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.718623 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fdh\" (UniqueName: \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.718882 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.718734 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.718940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.718870 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.719551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.719522 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.721243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.721224 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.730049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.730018 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fdh\" (UniqueName: \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\") pod \"maas-api-revoke-sub-64231135-5rpvf\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:52.824289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:52.824250 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:53.117973 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:53.117935 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ab3b90_619f_413a_a1c0_5f7d1940b296.slice/crio-eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525 WatchSource:0}: Error finding container eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525: Status 404 returned error can't find the container with id eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525 Oct 31 03:10:53.118256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.118230 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf"] Oct 31 03:10:53.122819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.122793 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:53.657528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.657478 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" event={"ID":"d0ab3b90-619f-413a-a1c0-5f7d1940b296","Type":"ContainerStarted","Data":"5c8e38377ea5ec90ba78bc2c5b4ac7f760c46a065267e4ac01ac5e4b7987ab3c"} Oct 31 03:10:53.657528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.657555 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" event={"ID":"d0ab3b90-619f-413a-a1c0-5f7d1940b296","Type":"ContainerStarted","Data":"eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525"} Oct 31 03:10:53.678893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.678827 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" podStartSLOduration=1.678808594 podStartE2EDuration="1.678808594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:53.673492795 +0000 UTC m=+931.528509937" watchObservedRunningTime="2026-10-31 03:10:53.678808594 +0000 UTC m=+931.533825734" Oct 31 03:10:53.907270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:53.907235 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:54.672097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:54.672062 4556 generic.go:356] "Generic (PLEG): container finished" podID="d0ab3b90-619f-413a-a1c0-5f7d1940b296" containerID="5c8e38377ea5ec90ba78bc2c5b4ac7f760c46a065267e4ac01ac5e4b7987ab3c" exitCode=0 Oct 31 03:10:54.672542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:54.672154 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" event={"ID":"d0ab3b90-619f-413a-a1c0-5f7d1940b296","Type":"ContainerDied","Data":"5c8e38377ea5ec90ba78bc2c5b4ac7f760c46a065267e4ac01ac5e4b7987ab3c"} Oct 31 03:10:54.772143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:54.772108 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:55.689012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.688952 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:55.805585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.805538 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf"] Oct 31 03:10:55.811597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.811567 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.856457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.856405 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf"] Oct 31 03:10:55.863857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.863214 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzfv\" (UniqueName: \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.863857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.863568 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.863857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.863792 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.905603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.905567 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:55.964982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.964879 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\" (UniqueName: \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\") pod \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " Oct 31 03:10:55.965155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965103 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\") pod \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " Oct 31 03:10:55.965228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965189 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\") pod \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\" (UID: \"d0ab3b90-619f-413a-a1c0-5f7d1940b296\") " Oct 31 03:10:55.965630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965599 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.965756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965621 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca" pod "d0ab3b90-619f-413a-a1c0-5f7d1940b296" (UID: "d0ab3b90-619f-413a-a1c0-5f7d1940b296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:10:55.965829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965753 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzfv\" (UniqueName: \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.965970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.965950 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.966060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.966043 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:55.966830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.966801 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.967667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.967644 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh" pod "d0ab3b90-619f-413a-a1c0-5f7d1940b296" (UID: "d0ab3b90-619f-413a-a1c0-5f7d1940b296"). InnerVolumeSpecName "kube-api-access-r2fdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:10:55.968486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.968398 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:55.968926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.968899 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca" pod "d0ab3b90-619f-413a-a1c0-5f7d1940b296" (UID: "d0ab3b90-619f-413a-a1c0-5f7d1940b296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:10:55.978108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:55.978067 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzfv\" (UniqueName: \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\") pod \"maas-api-revoke-sub-71077bf3-n66kf\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:56.068044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.068001 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ab3b90-619f-413a-a1c0-5f7d1940b296-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:56.068044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.068040 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2fdh\" (UniqueName: \"kubernetes.io/projected/d0ab3b90-619f-413a-a1c0-5f7d1940b296-kube-api-access-r2fdh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:10:56.130140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.130106 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:10:56.414451 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:56.414400 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033af525_76e5_4ab8_af2c_50685297b167.slice/crio-8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7 WatchSource:0}: Error finding container 8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7: Status 404 returned error can't find the container with id 8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7 Oct 31 03:10:56.414586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.414567 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf"] Oct 31 03:10:56.609004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.608975 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:56.707004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.706969 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" Oct 31 03:10:56.707520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.706989 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf" event={"ID":"d0ab3b90-619f-413a-a1c0-5f7d1940b296","Type":"ContainerDied","Data":"eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525"} Oct 31 03:10:56.707520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.707063 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eced7619b9f496ab8cef4309b5d8b94c6dd6aa9b1df61d6eaf35373836687525" Oct 31 03:10:56.709067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.708984 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" event={"ID":"033af525-76e5-4ab8-af2c-50685297b167","Type":"ContainerStarted","Data":"8f49bf615eb0d41cbc1ab97c00d9ae83bc03300ddda68837f33d80d705d6dd48"} Oct 31 03:10:56.709198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.709063 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" event={"ID":"033af525-76e5-4ab8-af2c-50685297b167","Type":"ContainerStarted","Data":"8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7"} Oct 31 03:10:56.740296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:56.740242 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" podStartSLOduration=1.740226636 podStartE2EDuration="1.740226636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:56.738689948 +0000 UTC m=+934.593707089" watchObservedRunningTime="2026-10-31 03:10:56.740226636 +0000 UTC m=+934.595243834" Oct 31 03:10:57.433012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:57.432967 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:57.718775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:57.718743 4556 generic.go:356] "Generic (PLEG): container finished" podID="033af525-76e5-4ab8-af2c-50685297b167" containerID="8f49bf615eb0d41cbc1ab97c00d9ae83bc03300ddda68837f33d80d705d6dd48" exitCode=0 Oct 31 03:10:57.719176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:57.718839 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" event={"ID":"033af525-76e5-4ab8-af2c-50685297b167","Type":"ContainerDied","Data":"8f49bf615eb0d41cbc1ab97c00d9ae83bc03300ddda68837f33d80d705d6dd48"} Oct 31 03:10:58.095511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.095422 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:10:58.096692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.096661 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d0ab3b90-619f-413a-a1c0-5f7d1940b296" containerName="revoke-keys" Oct 31 03:10:58.096866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.096844 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ab3b90-619f-413a-a1c0-5f7d1940b296" containerName="revoke-keys" Oct 31 03:10:58.097107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.097090 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ab3b90-619f-413a-a1c0-5f7d1940b296" containerName="revoke-keys" Oct 31 03:10:58.102457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.101188 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.138438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.135877 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:10:58.203920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.203870 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rq9\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.203939 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204081 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204138 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204243 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204483 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204543 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204573 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.204625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.204611 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.232710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.232666 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8"] Oct 31 03:10:58.238364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.238305 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.244567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.244533 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8"] Oct 31 03:10:58.284827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.284789 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:10:58.290293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.290265 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.322473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.322432 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323152 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323204 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.323334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323276 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323305 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323350 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323379 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6fq\" (UniqueName: \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.323649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323553 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rq9\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323605 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323648 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.323798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323688 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.323798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.323735 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.330010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.329981 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.331213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.331190 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.332151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.332127 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.332242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.332181 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:10:58.332524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.332502 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.333219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.333190 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.334237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.334203 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.345464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.338302 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.345464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.340296 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:10:58.345939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.345884 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.353256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.353224 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:10:58.374686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.374656 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.387276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.387194 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:58.394486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.394451 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rq9\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\") pod \"e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.429100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.429060 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.445120 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.445173 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.445254 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.445370 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446660 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446717 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446742 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446789 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446826 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6fq\" (UniqueName: \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446855 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhbxd\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.446938 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjn5c\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447123 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447193 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447428 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447462 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447520 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447648 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447695 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447778 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.447905 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.448009 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.459941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.449802 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.465251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.465215 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.467362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.467324 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6fq\" (UniqueName: \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\") pod \"maas-api-revoke-sub-c5168542-bdtd8\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.550836 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.550963 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551038 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551066 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551131 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551257 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551295 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551325 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551372 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551408 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhbxd\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551469 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjn5c\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551511 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551544 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551578 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551603 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551637 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551669 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.551876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.551707 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.552509 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.552771 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.553588 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.554242 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.554886 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.555138 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.555523 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.555865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.555703 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.555865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.555746 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.556503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.556472 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.556602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.556547 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.558009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.557971 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.558124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.558016 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.559482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.559395 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.561802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.561592 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:10:58.573449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.572496 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.582454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.575029 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.586173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.586133 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhbxd\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.588028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.588003 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjn5c\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\") pod \"e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.653551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.648618 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:10:58.780366 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.780053 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:10:58.978332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:58.978289 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:10:59.099396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.099350 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.099662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.099636 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.099847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.099818 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.150440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.149129 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8"] Oct 31 03:10:59.428353 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:10:59.428308 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1450a0ac_b5f7_4337_bee5_c50dde62649a.slice/crio-0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60 WatchSource:0}: Error finding container 0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60: Status 404 returned error can't find the container with id 0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60 Oct 31 03:10:59.444912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.443756 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:10:59.602060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.586307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:10:59.749079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.749032 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.750199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.750112 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.753820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.753786 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:10:59.904510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.903934 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" event={"ID":"92e097c0-a066-404d-ba85-709caa6abfb6","Type":"ContainerStarted","Data":"ef22ca57075544aeaad63e9b62c2558e89a28bd996cc1fad2ce872148c987e13"} Oct 31 03:10:59.907069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.906592 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" event={"ID":"1450a0ac-b5f7-4337-bee5-c50dde62649a","Type":"ContainerStarted","Data":"0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60"} Oct 31 03:10:59.908767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:10:59.908701 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" event={"ID":"f87f39e5-7c66-4bfe-ba30-8ebf168d5425","Type":"ContainerStarted","Data":"958ea14869c598ecca4a929aadab3d928618fe243d815323b2776e511a507722"} Oct 31 03:11:01.085504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.075254 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:11:01.155586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.155519 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.305517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.283652 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.333174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.332688 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:11:01.334100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.333964 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" event={"ID":"033af525-76e5-4ab8-af2c-50685297b167","Type":"ContainerDied","Data":"8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7"} Oct 31 03:11:01.334100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.334041 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8716f8dcf6b812a2c2853c771a7da6adc635d07b9cc6daf83a0d1bcac31675a7" Oct 31 03:11:01.363349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.363073 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:01.363349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.363170 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:01.363349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.363216 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:01.412844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.393311 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\" (UniqueName: \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\") pod \"033af525-76e5-4ab8-af2c-50685297b167\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " Oct 31 03:11:01.569729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.508830 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" event={"ID":"92e097c0-a066-404d-ba85-709caa6abfb6","Type":"ContainerStarted","Data":"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42"} Oct 31 03:11:01.569729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.537502 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:11:01.569729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.537669 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:01.569729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.537716 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:01.571069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.571033 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\") pod \"033af525-76e5-4ab8-af2c-50685297b167\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " Oct 31 03:11:01.577700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.572245 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\") pod \"033af525-76e5-4ab8-af2c-50685297b167\" (UID: \"033af525-76e5-4ab8-af2c-50685297b167\") " Oct 31 03:11:01.577700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.577648 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca" pod "033af525-76e5-4ab8-af2c-50685297b167" (UID: "033af525-76e5-4ab8-af2c-50685297b167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:01.689537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.590867 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv" pod "033af525-76e5-4ab8-af2c-50685297b167" (UID: "033af525-76e5-4ab8-af2c-50685297b167"). InnerVolumeSpecName "kube-api-access-2zzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:01.734794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.734748 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" event={"ID":"1450a0ac-b5f7-4337-bee5-c50dde62649a","Type":"ContainerStarted","Data":"966f0cd81e72cf668a31242a22995a552f308e27b2abe7ce4501f0f5fd539ae0"} Oct 31 03:11:01.867458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.867402 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033af525-76e5-4ab8-af2c-50685297b167-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:01.867953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.867929 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzfv\" (UniqueName: \"kubernetes.io/projected/033af525-76e5-4ab8-af2c-50685297b167-kube-api-access-2zzfv\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:01.926574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.926511 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.951764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.951639 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.956971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.956927 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:01.989440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.986113 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.020443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:01.996220 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.048562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.030030 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.071730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.050763 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca" pod "033af525-76e5-4ab8-af2c-50685297b167" (UID: "033af525-76e5-4ab8-af2c-50685297b167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:02.071730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.039440 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.128780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.108780 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033af525-76e5-4ab8-af2c-50685297b167-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:02.170853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.170794 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:02.171763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.171737 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:11:02.289502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.289155 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podStartSLOduration=4.289130876 podStartE2EDuration="4.289130876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:01.538530942 +0000 UTC m=+939.393548086" watchObservedRunningTime="2026-10-31 03:11:02.289130876 +0000 UTC m=+940.144148019" Oct 31 03:11:02.367442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.363688 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" podStartSLOduration=4.363659162 podStartE2EDuration="4.363659162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:01.890166989 +0000 UTC m=+939.745184132" watchObservedRunningTime="2026-10-31 03:11:02.363659162 +0000 UTC m=+940.218676304" Oct 31 03:11:02.494450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.492037 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:02.494450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.492203 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.494450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.493797 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.606496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.600835 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.656491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.648522 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:02.656491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.648597 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.682860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.682802 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.934554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.850148 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:02.934554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.886173 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:02.934554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.886235 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:02.934554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:02.886900 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:11:03.002839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.002800 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.044854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.014444 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.044854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.014765 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:03.044854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.015050 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:03.045267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.002683 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.101991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.096672 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.225986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.086094 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:03.225986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.218293 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:03.299467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.261856 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:03.299467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.261921 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358192 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358243 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358274 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358304 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358335 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358363 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358394 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:11:03.397605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.358495 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:11:03.533883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.447809 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:03.533883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.447867 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.828085 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt"] Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.829157 4556 cpu_manager.go:414] "containerMap: removing container" podUID="033af525-76e5-4ab8-af2c-50685297b167" containerName="revoke-keys" Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.829180 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033af525-76e5-4ab8-af2c-50685297b167" containerName="revoke-keys" Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.829396 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="033af525-76e5-4ab8-af2c-50685297b167" containerName="revoke-keys" Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.858662 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.931367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.869782 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:03.933064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.933010 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:04.097834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:03.951714 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.097834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.018866 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.097834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.020274 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.097834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.038005 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.154438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.140512 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:04.154438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.140696 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.160443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.158700 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:04.299112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.240774 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.299112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.273604 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.322622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.321579 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf" Oct 31 03:11:04.333825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.333688 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt"] Oct 31 03:11:04.489463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.448822 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:11:04.489463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.448892 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" event={"ID":"937e3608-09f9-41bc-9b8b-610e75b72540","Type":"ContainerStarted","Data":"901b7f2e77ab87d8503c678d96b4715761d457a481254102a9567231e390cf36"} Oct 31 03:11:04.489463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.334059 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.489463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.358868 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:04.514436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.501859 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.514436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.502045 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpl4\" (UniqueName: \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.514436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.502354 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.644291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.621928 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.644291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.622084 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpl4\" (UniqueName: \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.644291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.622317 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.719405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.707929 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:04.843270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.752526 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:04.843270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:04.752586 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:05.252439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.251220 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:05.336792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.313531 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.362776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.338810 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.417631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.375819 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.417631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.378727 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:05.519723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.515957 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpl4\" (UniqueName: \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\") pod \"maas-api-revoke-sub-224ab914-67fzt\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:05.519723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.519514 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.532657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.520231 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:05.532657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.521569 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:05.532657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.530617 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.538739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.538692 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.559535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.559505 4556 trace.go:236] Trace[744589245]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-57b855755-jfdnq" (31-Oct-2026 03:11:04.286) (total time: 1273ms): Oct 31 03:11:05.559535 ip-10-0-139-52 kubenswrapper[4556]: Trace[744589245]: [1.273152838s] [1.273152838s] END Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.638003 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.638090 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.664692 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ztdt5" podUID="377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:05.666840 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.218s" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.666911 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.667018 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.667138 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.667207 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.671545 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.671866 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.672109 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.676487 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.682368 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.684940 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.684991 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.685443 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:11:05.748645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.685935 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:05.802673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.750718 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:05.802673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.750777 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:05.993434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:05.985853 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:11:06.067390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.067284 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:11:06.067985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.067677 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:11:06.148522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.148364 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:11:06.477502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.366515 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.629440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.591949 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:11:06.629440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.592264 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:06.659004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.658965 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.709570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.689537 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:06.709819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.709595 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:06.709958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.709946 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:11:06.710042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.665641 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podStartSLOduration=8.665619279 podStartE2EDuration="8.665619279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:04.385114674 +0000 UTC m=+942.240131826" watchObservedRunningTime="2026-10-31 03:11:06.665619279 +0000 UTC m=+944.520636421" Oct 31 03:11:06.860542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.709647 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.861034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.709682 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:06.861478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.764845 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.861840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.766139 4556 generic.go:356] "Generic (PLEG): container finished" podID="1450a0ac-b5f7-4337-bee5-c50dde62649a" containerID="966f0cd81e72cf668a31242a22995a552f308e27b2abe7ce4501f0f5fd539ae0" exitCode=0 Oct 31 03:11:06.862288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.766940 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:06.862516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.800964 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" event={"ID":"1450a0ac-b5f7-4337-bee5-c50dde62649a","Type":"ContainerDied","Data":"966f0cd81e72cf668a31242a22995a552f308e27b2abe7ce4501f0f5fd539ae0"} Oct 31 03:11:06.862799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.855504 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:06.862933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.855826 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:06.863060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.863034 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:06.991448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.936121 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" event={"ID":"f87f39e5-7c66-4bfe-ba30-8ebf168d5425","Type":"ContainerStarted","Data":"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018"} Oct 31 03:11:07.023824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:06.995514 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr"] Oct 31 03:11:07.154503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.065444 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" event={"ID":"937e3608-09f9-41bc-9b8b-610e75b72540","Type":"ContainerStarted","Data":"6955428a78ffcf1072fdabc05802bd2a02b123f9306975a8ea59a2006861cbbb"} Oct 31 03:11:07.154827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.154791 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:11:07.158191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.108759 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.165240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.165216 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.269766 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.215457 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.271087 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.271133 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjt4\" (UniqueName: \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.412840 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.413097 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.413132 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjt4\" (UniqueName: \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.433204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.414885 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:07.456912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.441561 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr"] Oct 31 03:11:07.456912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.454851 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:07.540869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.540834 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.566615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.541393 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:07.599210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.599154 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:07.650754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.650696 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:07.691589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.689746 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:07.813609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.813549 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.813884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.813851 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:07.827470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.814268 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:11:07.953880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.816534 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:07.954042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.953964 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:07.954042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.933065 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 31 03:11:07.954162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.933099 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:07.968452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:07.932875 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:08.061491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.061443 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": context deadline exceeded" Oct 31 03:11:08.063592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.063543 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:11:08.087498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.087464 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:11:08.240683 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:08.224203 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2068dd00_6f30_464c_9d62_cd7d38956bcb.slice/crio-f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3 WatchSource:0}: Error finding container f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3: Status 404 returned error can't find the container with id f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3 Oct 31 03:11:08.240683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.225904 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:11:08.351444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.343401 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:08.420382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.352651 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:08.526497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.523519 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjt4\" (UniqueName: \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\") pod \"maas-api-revoke-sub-079550a4-9glkr\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:08.716441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.712193 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.716441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.712285 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:08.804764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.804663 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.962463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.961764 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:08.962463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.941763 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:08.962778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.840252 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:08.986747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.945945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:08.986747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.867118 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:08.986747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.970623 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:08.986747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.941911 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:08.990099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:08.987365 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:09.207345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.199154 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:09.349507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.333589 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:09.571440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.568882 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.571440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.568985 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:09.626494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.588033 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:09.660190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.660157 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.660460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.660399 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:09.736980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.727709 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:09.797173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.797113 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" event={"ID":"2068dd00-6f30-464c-9d62-cd7d38956bcb","Type":"ContainerStarted","Data":"f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3"} Oct 31 03:11:09.804549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.804501 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:09.804789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.804756 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:09.805977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.805838 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:09.806906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.806865 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:09.807353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.807328 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 31 03:11:09.864138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:09.861385 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 31 03:11:10.362995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.362909 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podStartSLOduration=12.362889342 podStartE2EDuration="12.362889342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:07.238313305 +0000 UTC m=+945.093330461" watchObservedRunningTime="2026-10-31 03:11:10.362889342 +0000 UTC m=+948.217906484" Oct 31 03:11:10.442899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.441925 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 03:11:10.442899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.441989 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 03:11:10.549437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.544774 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:10.685529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.685485 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:10.685707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.685552 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:10.794133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.794091 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:10.829556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.829499 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:10.864639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.864595 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:11:10.865065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.865035 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" event={"ID":"1450a0ac-b5f7-4337-bee5-c50dde62649a","Type":"ContainerDied","Data":"0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60"} Oct 31 03:11:10.865174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.865093 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0760cccf86867c9d8a47296c0008f0363870627d75cec53051506e794f815e60" Oct 31 03:11:10.898324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.893546 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt"] Oct 31 03:11:10.961765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.961730 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\") pod \"1450a0ac-b5f7-4337-bee5-c50dde62649a\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " Oct 31 03:11:10.962035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.962008 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\") pod \"1450a0ac-b5f7-4337-bee5-c50dde62649a\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " Oct 31 03:11:10.962218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.962189 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\" (UniqueName: \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\") pod \"1450a0ac-b5f7-4337-bee5-c50dde62649a\" (UID: \"1450a0ac-b5f7-4337-bee5-c50dde62649a\") " Oct 31 03:11:10.963716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.963688 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca" pod "1450a0ac-b5f7-4337-bee5-c50dde62649a" (UID: "1450a0ac-b5f7-4337-bee5-c50dde62649a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:11.000435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:10.991552 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca" pod "1450a0ac-b5f7-4337-bee5-c50dde62649a" (UID: "1450a0ac-b5f7-4337-bee5-c50dde62649a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:11.011435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.010596 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq" pod "1450a0ac-b5f7-4337-bee5-c50dde62649a" (UID: "1450a0ac-b5f7-4337-bee5-c50dde62649a"). InnerVolumeSpecName "kube-api-access-hp6fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:11.081448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.071626 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6fq\" (UniqueName: \"kubernetes.io/projected/1450a0ac-b5f7-4337-bee5-c50dde62649a-kube-api-access-hp6fq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:11.081448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.071674 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:11.081448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.071707 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1450a0ac-b5f7-4337-bee5-c50dde62649a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:11.124789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.124195 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr"] Oct 31 03:11:11.445196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.445154 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:11.650087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.650046 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:11:11.650282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.650116 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:11:11.785471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.782242 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:11.785471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.782312 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:11.884124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.883952 4556 generic.go:356] "Generic (PLEG): container finished" podID="2068dd00-6f30-464c-9d62-cd7d38956bcb" containerID="6013b1fd1b96524c24037c33ba19deb5a4b18dc2f1746f8046a4b257fbb05289" exitCode=0 Oct 31 03:11:11.884124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.884036 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" event={"ID":"2068dd00-6f30-464c-9d62-cd7d38956bcb","Type":"ContainerDied","Data":"6013b1fd1b96524c24037c33ba19deb5a4b18dc2f1746f8046a4b257fbb05289"} Oct 31 03:11:11.888756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.888725 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8" Oct 31 03:11:11.888954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.888835 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" event={"ID":"855e6d0e-49f1-4ed1-9b91-80b61e661c04","Type":"ContainerStarted","Data":"b7e73d6ade03d09446099f3805280ea5a97406b4f67b0dfe509f82420b9bff3c"} Oct 31 03:11:11.888954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.888890 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" event={"ID":"855e6d0e-49f1-4ed1-9b91-80b61e661c04","Type":"ContainerStarted","Data":"01db6feed48e03fa94983ccbc40951f60709aea80e2729a98362dde9e243520f"} Oct 31 03:11:11.901632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.901557 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:11:11.903637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.903613 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:11:11.917004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.916961 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:11:11.925667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:11.924532 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" podStartSLOduration=7.92451079 podStartE2EDuration="7.92451079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:11.920180244 +0000 UTC m=+949.775197386" watchObservedRunningTime="2026-10-31 03:11:11.92451079 +0000 UTC m=+949.779527934" Oct 31 03:11:12.215942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.215844 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:12.735622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.735583 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:11:12.736054 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.736033 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:11:12.736947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.736921 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:11:12.781367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.781319 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:11:12.781573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.781388 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 31 03:11:12.916833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.916789 4556 generic.go:356] "Generic (PLEG): container finished" podID="855e6d0e-49f1-4ed1-9b91-80b61e661c04" containerID="b7e73d6ade03d09446099f3805280ea5a97406b4f67b0dfe509f82420b9bff3c" exitCode=0 Oct 31 03:11:12.917403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:12.916866 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" event={"ID":"855e6d0e-49f1-4ed1-9b91-80b61e661c04","Type":"ContainerDied","Data":"b7e73d6ade03d09446099f3805280ea5a97406b4f67b0dfe509f82420b9bff3c"} Oct 31 03:11:13.125450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.119702 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:13.224316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.224284 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:13.377729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.377636 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\") pod \"2068dd00-6f30-464c-9d62-cd7d38956bcb\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " Oct 31 03:11:13.378042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.378009 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\" (UniqueName: \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\") pod \"2068dd00-6f30-464c-9d62-cd7d38956bcb\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " Oct 31 03:11:13.378126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.378057 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca" pod "2068dd00-6f30-464c-9d62-cd7d38956bcb" (UID: "2068dd00-6f30-464c-9d62-cd7d38956bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:13.378170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.378159 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\") pod \"2068dd00-6f30-464c-9d62-cd7d38956bcb\" (UID: \"2068dd00-6f30-464c-9d62-cd7d38956bcb\") " Oct 31 03:11:13.378898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.378877 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:13.380186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.380164 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca" pod "2068dd00-6f30-464c-9d62-cd7d38956bcb" (UID: "2068dd00-6f30-464c-9d62-cd7d38956bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:13.380429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.380390 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4" pod "2068dd00-6f30-464c-9d62-cd7d38956bcb" (UID: "2068dd00-6f30-464c-9d62-cd7d38956bcb"). InnerVolumeSpecName "kube-api-access-6tpl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:13.480895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.480858 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpl4\" (UniqueName: \"kubernetes.io/projected/2068dd00-6f30-464c-9d62-cd7d38956bcb-kube-api-access-6tpl4\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:13.480895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.480894 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2068dd00-6f30-464c-9d62-cd7d38956bcb-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:13.781908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.781874 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:11:13.786864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.786833 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:11:13.937865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.937643 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" event={"ID":"2068dd00-6f30-464c-9d62-cd7d38956bcb","Type":"ContainerDied","Data":"f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3"} Oct 31 03:11:13.937865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.937716 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c3496b55d00c36842900b7a8b6525571f725cd0671e1c45a91987b9f990cc3" Oct 31 03:11:13.937865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.937733 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt" Oct 31 03:11:13.940290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:13.940260 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:11:14.035855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.035764 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:14.160736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.160705 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:14.305342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.305252 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\" (UniqueName: \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\") pod \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " Oct 31 03:11:14.305342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.305312 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\") pod \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " Oct 31 03:11:14.305665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.305641 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\") pod \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\" (UID: \"855e6d0e-49f1-4ed1-9b91-80b61e661c04\") " Oct 31 03:11:14.305722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.305680 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca" pod "855e6d0e-49f1-4ed1-9b91-80b61e661c04" (UID: "855e6d0e-49f1-4ed1-9b91-80b61e661c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:14.306251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.306231 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:14.307501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.307466 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4" pod "855e6d0e-49f1-4ed1-9b91-80b61e661c04" (UID: "855e6d0e-49f1-4ed1-9b91-80b61e661c04"). InnerVolumeSpecName "kube-api-access-8xjt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:14.307595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.307565 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca" pod "855e6d0e-49f1-4ed1-9b91-80b61e661c04" (UID: "855e6d0e-49f1-4ed1-9b91-80b61e661c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:14.408353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.408315 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/855e6d0e-49f1-4ed1-9b91-80b61e661c04-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:14.408353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.408344 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjt4\" (UniqueName: \"kubernetes.io/projected/855e6d0e-49f1-4ed1-9b91-80b61e661c04-kube-api-access-8xjt4\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:14.896759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.896724 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:14.948808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.948779 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" Oct 31 03:11:14.949534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.949490 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr" event={"ID":"855e6d0e-49f1-4ed1-9b91-80b61e661c04","Type":"ContainerDied","Data":"01db6feed48e03fa94983ccbc40951f60709aea80e2729a98362dde9e243520f"} Oct 31 03:11:14.949644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.949554 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01db6feed48e03fa94983ccbc40951f60709aea80e2729a98362dde9e243520f" Oct 31 03:11:14.984648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.984614 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6"] Oct 31 03:11:14.985295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985280 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2068dd00-6f30-464c-9d62-cd7d38956bcb" containerName="revoke-keys" Oct 31 03:11:14.985343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985297 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2068dd00-6f30-464c-9d62-cd7d38956bcb" containerName="revoke-keys" Oct 31 03:11:14.985343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985340 4556 cpu_manager.go:414] "containerMap: removing container" podUID="855e6d0e-49f1-4ed1-9b91-80b61e661c04" containerName="revoke-keys" Oct 31 03:11:14.985409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985345 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855e6d0e-49f1-4ed1-9b91-80b61e661c04" containerName="revoke-keys" Oct 31 03:11:14.985409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985365 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1450a0ac-b5f7-4337-bee5-c50dde62649a" containerName="revoke-keys" Oct 31 03:11:14.985409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985370 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1450a0ac-b5f7-4337-bee5-c50dde62649a" containerName="revoke-keys" Oct 31 03:11:14.985524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985513 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="855e6d0e-49f1-4ed1-9b91-80b61e661c04" containerName="revoke-keys" Oct 31 03:11:14.985555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985531 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1450a0ac-b5f7-4337-bee5-c50dde62649a" containerName="revoke-keys" Oct 31 03:11:14.985555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.985537 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2068dd00-6f30-464c-9d62-cd7d38956bcb" containerName="revoke-keys" Oct 31 03:11:14.989903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.989869 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:14.993590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.993545 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:11:14.994480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:14.994444 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:11:15.001656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.001626 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6"] Oct 31 03:11:15.017767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.016544 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gl9\" (UniqueName: \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.017767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.016833 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.017767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.016884 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.118021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.117976 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.118217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.118055 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.118217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.118183 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gl9\" (UniqueName: \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.118652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.118628 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.120473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.120451 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.129704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.129678 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gl9\" (UniqueName: \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\") pod \"maas-api-revoke-sub-a4007e31-cczz6\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.312901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.312859 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:15.602668 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:15.602637 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b68351c_1856_4feb_8864_63b291e769f7.slice/crio-543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d WatchSource:0}: Error finding container 543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d: Status 404 returned error can't find the container with id 543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d Oct 31 03:11:15.604499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.604470 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6"] Oct 31 03:11:15.738748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.738699 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:15.938371 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:15.938329 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:11:15.957567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.957532 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" event={"ID":"8b68351c-1856-4feb-8864-63b291e769f7","Type":"ContainerStarted","Data":"04a9b25ff1c8dc3550a68e71807a85e752079efd60724cb98a0397e4172ea848"} Oct 31 03:11:15.957936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.957596 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" event={"ID":"8b68351c-1856-4feb-8864-63b291e769f7","Type":"ContainerStarted","Data":"543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d"} Oct 31 03:11:15.978837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:15.978710 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" podStartSLOduration=1.9786931540000001 podStartE2EDuration="1.978693154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:15.975555418 +0000 UTC m=+953.830572572" watchObservedRunningTime="2026-10-31 03:11:15.978693154 +0000 UTC m=+953.833710313" Oct 31 03:11:16.526680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:16.526643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:16.969349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:16.969317 4556 generic.go:356] "Generic (PLEG): container finished" podID="8b68351c-1856-4feb-8864-63b291e769f7" containerID="04a9b25ff1c8dc3550a68e71807a85e752079efd60724cb98a0397e4172ea848" exitCode=0 Oct 31 03:11:16.969759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:16.969390 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" event={"ID":"8b68351c-1856-4feb-8864-63b291e769f7","Type":"ContainerDied","Data":"04a9b25ff1c8dc3550a68e71807a85e752079efd60724cb98a0397e4172ea848"} Oct 31 03:11:17.426270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:17.426225 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:18.012667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.012627 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g"] Oct 31 03:11:18.017611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.017569 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.031170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.031136 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g"] Oct 31 03:11:18.068473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.065496 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.068473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.065577 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdhc\" (UniqueName: \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.068473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.065666 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.147619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.147558 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:18.167222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.167185 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.167400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.167263 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdhc\" (UniqueName: \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.167550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.167406 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.168495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.168462 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.174685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.174622 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.197849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.196764 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdhc\" (UniqueName: \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\") pod \"maas-api-revoke-sub-2778e280-9ws5g\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.231980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.231906 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:18.270114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.270075 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\") pod \"8b68351c-1856-4feb-8864-63b291e769f7\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " Oct 31 03:11:18.270281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.270160 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\" (UniqueName: \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\") pod \"8b68351c-1856-4feb-8864-63b291e769f7\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " Oct 31 03:11:18.270281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.270269 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\") pod \"8b68351c-1856-4feb-8864-63b291e769f7\" (UID: \"8b68351c-1856-4feb-8864-63b291e769f7\") " Oct 31 03:11:18.270903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.270835 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca" pod "8b68351c-1856-4feb-8864-63b291e769f7" (UID: "8b68351c-1856-4feb-8864-63b291e769f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:18.271533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.271256 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:18.273511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.273479 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9" pod "8b68351c-1856-4feb-8864-63b291e769f7" (UID: "8b68351c-1856-4feb-8864-63b291e769f7"). InnerVolumeSpecName "kube-api-access-m9gl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:18.273638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.273509 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca" pod "8b68351c-1856-4feb-8864-63b291e769f7" (UID: "8b68351c-1856-4feb-8864-63b291e769f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:18.360368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.360330 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:18.373487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.373332 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b68351c-1856-4feb-8864-63b291e769f7-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:18.373487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.373383 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gl9\" (UniqueName: \"kubernetes.io/projected/8b68351c-1856-4feb-8864-63b291e769f7-kube-api-access-m9gl9\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:18.653442 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:18.653390 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dca2a48_ca2d_4ada_9468_e2fe881bc5d9.slice/crio-3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5 WatchSource:0}: Error finding container 3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5: Status 404 returned error can't find the container with id 3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5 Oct 31 03:11:18.655562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.655543 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g"] Oct 31 03:11:18.943308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.943220 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw"] Oct 31 03:11:18.944005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.943985 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8b68351c-1856-4feb-8864-63b291e769f7" containerName="revoke-keys" Oct 31 03:11:18.944055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.944006 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b68351c-1856-4feb-8864-63b291e769f7" containerName="revoke-keys" Oct 31 03:11:18.944158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.944141 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b68351c-1856-4feb-8864-63b291e769f7" containerName="revoke-keys" Oct 31 03:11:18.949400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.949368 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:18.955271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.955240 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw"] Oct 31 03:11:18.982178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.982144 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nz4\" (UniqueName: \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:18.982294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.982247 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:18.982469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.982449 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:18.991492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.991463 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" Oct 31 03:11:18.991492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.991478 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6" event={"ID":"8b68351c-1856-4feb-8864-63b291e769f7","Type":"ContainerDied","Data":"543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d"} Oct 31 03:11:18.991725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.991546 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543a1c56f4c30e4b8661a64a2cb6c25a2a6c3a437bacc31792da7926c3e0a66d" Oct 31 03:11:18.994080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:18.994051 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" event={"ID":"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9","Type":"ContainerStarted","Data":"3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5"} Oct 31 03:11:19.001628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.001602 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:19.084484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.084429 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nz4\" (UniqueName: \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.087281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.084545 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.087281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.085218 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.087281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.085462 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.088888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.088861 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.098590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.098561 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nz4\" (UniqueName: \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\") pod \"maas-api-revoke-sub-4b7c38df-dhknw\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.267148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.267066 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:19.363374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.363333 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79"] Oct 31 03:11:19.369358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.369320 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.379083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.379047 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79"] Oct 31 03:11:19.494373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.494334 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwc6m\" (UniqueName: \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.494589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.494399 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.494589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.494545 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.552561 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:19.552522 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0807b4bc_d0c9_4008_9017_c5032e50f789.slice/crio-29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66 WatchSource:0}: Error finding container 29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66: Status 404 returned error can't find the container with id 29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66 Oct 31 03:11:19.553805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.553776 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw"] Oct 31 03:11:19.596527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.596489 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwc6m\" (UniqueName: \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.596679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.596569 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.596679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.596641 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.597348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.597310 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.598985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.598960 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.608239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.608216 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwc6m\" (UniqueName: \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\") pod \"maas-api-revoke-sub-7a7aecd0-7hn79\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.693674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.693634 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:19.912702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.912664 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:19.986833 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:19.986799 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4954a30b_9d8b_4416_9954_e6ede8f88354.slice/crio-74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba WatchSource:0}: Error finding container 74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba: Status 404 returned error can't find the container with id 74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba Oct 31 03:11:19.991302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:19.991269 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79"] Oct 31 03:11:20.005894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:20.005859 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" event={"ID":"0807b4bc-d0c9-4008-9017-c5032e50f789","Type":"ContainerStarted","Data":"29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66"} Oct 31 03:11:20.007839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:20.007809 4556 generic.go:356] "Generic (PLEG): container finished" podID="9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" containerID="045856e80ce5428ccdec2811c5aac79092e71b1bfb9dd64e9692aa7868c50d2b" exitCode=0 Oct 31 03:11:20.007980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:20.007842 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" event={"ID":"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9","Type":"ContainerDied","Data":"045856e80ce5428ccdec2811c5aac79092e71b1bfb9dd64e9692aa7868c50d2b"} Oct 31 03:11:20.009564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:20.009537 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" event={"ID":"4954a30b-9d8b-4416-9954-e6ede8f88354","Type":"ContainerStarted","Data":"74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba"} Oct 31 03:11:20.735667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:20.735632 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:21.020689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.020652 4556 generic.go:356] "Generic (PLEG): container finished" podID="0807b4bc-d0c9-4008-9017-c5032e50f789" containerID="e82aa974704567cafb2e8736eaccfc68ea35f08b55882c5fa509ca15acb14958" exitCode=0 Oct 31 03:11:21.020864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.020731 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" event={"ID":"0807b4bc-d0c9-4008-9017-c5032e50f789","Type":"ContainerDied","Data":"e82aa974704567cafb2e8736eaccfc68ea35f08b55882c5fa509ca15acb14958"} Oct 31 03:11:21.022686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.022660 4556 generic.go:356] "Generic (PLEG): container finished" podID="4954a30b-9d8b-4416-9954-e6ede8f88354" containerID="478ee0c8efcc55f69f27f203d739957490235147cc054ffaefed51098b370508" exitCode=0 Oct 31 03:11:21.022811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.022712 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" event={"ID":"4954a30b-9d8b-4416-9954-e6ede8f88354","Type":"ContainerDied","Data":"478ee0c8efcc55f69f27f203d739957490235147cc054ffaefed51098b370508"} Oct 31 03:11:21.193362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.193319 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:21.327158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.327025 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\" (UniqueName: \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\") pod \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " Oct 31 03:11:21.327341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.327229 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\") pod \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " Oct 31 03:11:21.327453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.327434 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\") pod \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\" (UID: \"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9\") " Oct 31 03:11:21.330140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.330102 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca" pod "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" (UID: "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:21.332114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.332079 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca" pod "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" (UID: "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:21.332241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.332117 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc" pod "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" (UID: "9dca2a48-ca2d-4ada-9468-e2fe881bc5d9"). InnerVolumeSpecName "kube-api-access-jsdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:21.429797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.429760 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:21.429797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.429793 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:21.430003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.429807 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsdhc\" (UniqueName: \"kubernetes.io/projected/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9-kube-api-access-jsdhc\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:21.515585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.515545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:21.956679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.956638 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8"] Oct 31 03:11:21.957316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.957300 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" containerName="revoke-keys" Oct 31 03:11:21.957316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.957317 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" containerName="revoke-keys" Oct 31 03:11:21.957506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.957483 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" containerName="revoke-keys" Oct 31 03:11:21.961680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.961659 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:21.972046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:21.972013 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8"] Oct 31 03:11:22.037962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.037926 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" Oct 31 03:11:22.038171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.037932 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g" event={"ID":"9dca2a48-ca2d-4ada-9468-e2fe881bc5d9","Type":"ContainerDied","Data":"3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5"} Oct 31 03:11:22.038171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.038088 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d7b140254e72a4999573ca25932b7ac5c59bac678233c6f07ec3925aae51fb5" Oct 31 03:11:22.041562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.041522 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.041726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.041615 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwpl\" (UniqueName: \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.041726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.041668 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.143277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.143239 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.143453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.143341 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwpl\" (UniqueName: \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.143453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.143408 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.144201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.144165 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.146591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.146563 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.157868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.156726 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwpl\" (UniqueName: \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\") pod \"maas-api-revoke-sub-1c253abb-24kr8\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.259242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.259209 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:22.259392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.259222 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:22.288087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.288050 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.352805 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\") pod \"4954a30b-9d8b-4416-9954-e6ede8f88354\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.352948 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\") pod \"0807b4bc-d0c9-4008-9017-c5032e50f789\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.352992 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\") pod \"0807b4bc-d0c9-4008-9017-c5032e50f789\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.353037 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\" (UniqueName: \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\") pod \"0807b4bc-d0c9-4008-9017-c5032e50f789\" (UID: \"0807b4bc-d0c9-4008-9017-c5032e50f789\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.353198 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\") pod \"4954a30b-9d8b-4416-9954-e6ede8f88354\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.353231 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.353326 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\" (UniqueName: \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\") pod \"4954a30b-9d8b-4416-9954-e6ede8f88354\" (UID: \"4954a30b-9d8b-4416-9954-e6ede8f88354\") " Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.353892 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca" pod "0807b4bc-d0c9-4008-9017-c5032e50f789" (UID: "0807b4bc-d0c9-4008-9017-c5032e50f789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.354189 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.354968 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca" pod "4954a30b-9d8b-4416-9954-e6ede8f88354" (UID: "4954a30b-9d8b-4416-9954-e6ede8f88354"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.360132 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4" pod "0807b4bc-d0c9-4008-9017-c5032e50f789" (UID: "0807b4bc-d0c9-4008-9017-c5032e50f789"). InnerVolumeSpecName "kube-api-access-b9nz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.360438 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca" pod "4954a30b-9d8b-4416-9954-e6ede8f88354" (UID: "4954a30b-9d8b-4416-9954-e6ede8f88354"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.362854 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca" pod "0807b4bc-d0c9-4008-9017-c5032e50f789" (UID: "0807b4bc-d0c9-4008-9017-c5032e50f789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:22.370662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.369932 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m" pod "4954a30b-9d8b-4416-9954-e6ede8f88354" (UID: "4954a30b-9d8b-4416-9954-e6ede8f88354"). InnerVolumeSpecName "kube-api-access-vwc6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:22.464852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.464812 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.464852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.464850 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwc6m\" (UniqueName: \"kubernetes.io/projected/4954a30b-9d8b-4416-9954-e6ede8f88354-kube-api-access-vwc6m\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.465110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.464866 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4954a30b-9d8b-4416-9954-e6ede8f88354-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.465110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.464880 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0807b4bc-d0c9-4008-9017-c5032e50f789-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.465110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.464896 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nz4\" (UniqueName: \"kubernetes.io/projected/0807b4bc-d0c9-4008-9017-c5032e50f789-kube-api-access-b9nz4\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:22.679850 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:22.679813 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c199759_0aca_46cf_a4b6_38f5c32ec4d1.slice/crio-c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa WatchSource:0}: Error finding container c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa: Status 404 returned error can't find the container with id c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa Oct 31 03:11:22.681186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:22.681161 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8"] Oct 31 03:11:23.051708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.051673 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" Oct 31 03:11:23.051708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.051704 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79" event={"ID":"4954a30b-9d8b-4416-9954-e6ede8f88354","Type":"ContainerDied","Data":"74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba"} Oct 31 03:11:23.052228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.051754 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d369ed2a00ac9c54698da4bb4c673add887b4bb4ca791ba5ae0ebdd29643ba" Oct 31 03:11:23.053304 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.053281 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" Oct 31 03:11:23.053462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.053280 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw" event={"ID":"0807b4bc-d0c9-4008-9017-c5032e50f789","Type":"ContainerDied","Data":"29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66"} Oct 31 03:11:23.053462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.053393 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c9a4c040490330bc8b0e45675a06957f41a8b9d4ac4db371c6a62392c68a66" Oct 31 03:11:23.055206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.055177 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" event={"ID":"6c199759-0aca-46cf-a4b6-38f5c32ec4d1","Type":"ContainerStarted","Data":"d46d97c16d3491124dd3fb990523c540ac90fb692c35284444712e276e9ae785"} Oct 31 03:11:23.055315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.055230 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" event={"ID":"6c199759-0aca-46cf-a4b6-38f5c32ec4d1","Type":"ContainerStarted","Data":"c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa"} Oct 31 03:11:23.073096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.073028 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" podStartSLOduration=2.073009201 podStartE2EDuration="2.073009201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:23.071656599 +0000 UTC m=+960.926673742" watchObservedRunningTime="2026-10-31 03:11:23.073009201 +0000 UTC m=+960.928026343" Oct 31 03:11:23.330899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.330792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:23.823692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.823653 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt"] Oct 31 03:11:23.824372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824355 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4954a30b-9d8b-4416-9954-e6ede8f88354" containerName="revoke-keys" Oct 31 03:11:23.824425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824374 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4954a30b-9d8b-4416-9954-e6ede8f88354" containerName="revoke-keys" Oct 31 03:11:23.824425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824393 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0807b4bc-d0c9-4008-9017-c5032e50f789" containerName="revoke-keys" Oct 31 03:11:23.824425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824399 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0807b4bc-d0c9-4008-9017-c5032e50f789" containerName="revoke-keys" Oct 31 03:11:23.824550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824539 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4954a30b-9d8b-4416-9954-e6ede8f88354" containerName="revoke-keys" Oct 31 03:11:23.824581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.824555 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0807b4bc-d0c9-4008-9017-c5032e50f789" containerName="revoke-keys" Oct 31 03:11:23.828475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.828456 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:23.846258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.846226 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt"] Oct 31 03:11:23.990035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.989999 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:23.990282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.990121 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:23.990338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:23.990326 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7gg\" (UniqueName: \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.065085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.065050 4556 generic.go:356] "Generic (PLEG): container finished" podID="6c199759-0aca-46cf-a4b6-38f5c32ec4d1" containerID="d46d97c16d3491124dd3fb990523c540ac90fb692c35284444712e276e9ae785" exitCode=0 Oct 31 03:11:24.065535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.065131 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" event={"ID":"6c199759-0aca-46cf-a4b6-38f5c32ec4d1","Type":"ContainerDied","Data":"d46d97c16d3491124dd3fb990523c540ac90fb692c35284444712e276e9ae785"} Oct 31 03:11:24.091303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.091206 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7gg\" (UniqueName: \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.091303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.091280 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.091585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.091377 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.092209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.092176 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.093761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.093742 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.102656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.102623 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7gg\" (UniqueName: \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\") pod \"maas-api-revoke-sub-33d3fc93-zgdpt\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.146906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.146870 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:24.172233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.172199 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:24.465462 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:24.465430 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f6ccbb_febc_4cfb_95c5_3c5169aa65f5.slice/crio-85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad WatchSource:0}: Error finding container 85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad: Status 404 returned error can't find the container with id 85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad Oct 31 03:11:24.467128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.467102 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt"] Oct 31 03:11:24.906658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:24.905623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:25.076076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.075981 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" event={"ID":"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5","Type":"ContainerStarted","Data":"dd4f9f13f7aa37d8c11ebfcb41335cc2fc8cb2b664cba78e19a6fdbccc28b5dc"} Oct 31 03:11:25.076076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.076033 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" event={"ID":"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5","Type":"ContainerStarted","Data":"85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad"} Oct 31 03:11:25.099852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.099792 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" podStartSLOduration=2.099775015 podStartE2EDuration="2.099775015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:25.094455614 +0000 UTC m=+962.949472755" watchObservedRunningTime="2026-10-31 03:11:25.099775015 +0000 UTC m=+962.954792180" Oct 31 03:11:25.241128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.241103 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:25.416996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.416964 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\" (UniqueName: \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\") pod \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " Oct 31 03:11:25.417200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.417067 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\") pod \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " Oct 31 03:11:25.417200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.417192 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\") pod \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\" (UID: \"6c199759-0aca-46cf-a4b6-38f5c32ec4d1\") " Oct 31 03:11:25.417528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.417496 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca" pod "6c199759-0aca-46cf-a4b6-38f5c32ec4d1" (UID: "6c199759-0aca-46cf-a4b6-38f5c32ec4d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:25.418394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.418165 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:25.419603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.419576 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca" pod "6c199759-0aca-46cf-a4b6-38f5c32ec4d1" (UID: "6c199759-0aca-46cf-a4b6-38f5c32ec4d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:25.419745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.419619 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl" pod "6c199759-0aca-46cf-a4b6-38f5c32ec4d1" (UID: "6c199759-0aca-46cf-a4b6-38f5c32ec4d1"). InnerVolumeSpecName "kube-api-access-drwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:25.519627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.519587 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwpl\" (UniqueName: \"kubernetes.io/projected/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-kube-api-access-drwpl\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:25.519627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.519624 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c199759-0aca-46cf-a4b6-38f5c32ec4d1-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:25.645782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:25.645689 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:26.087492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.087445 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" event={"ID":"6c199759-0aca-46cf-a4b6-38f5c32ec4d1","Type":"ContainerDied","Data":"c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa"} Oct 31 03:11:26.087492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.087484 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8" Oct 31 03:11:26.088018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.087514 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2dda6e02c0cfa52a69548829e654ac7a0fb524ce0786db27459ed0d56bef6aa" Oct 31 03:11:26.089624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.089593 4556 generic.go:356] "Generic (PLEG): container finished" podID="98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" containerID="dd4f9f13f7aa37d8c11ebfcb41335cc2fc8cb2b664cba78e19a6fdbccc28b5dc" exitCode=0 Oct 31 03:11:26.089753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.089676 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" event={"ID":"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5","Type":"ContainerDied","Data":"dd4f9f13f7aa37d8c11ebfcb41335cc2fc8cb2b664cba78e19a6fdbccc28b5dc"} Oct 31 03:11:26.610736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:26.610682 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:27.258212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.258184 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:27.352008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.351917 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\") pod \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " Oct 31 03:11:27.352185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.352011 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\") pod \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " Oct 31 03:11:27.352277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.352249 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\" (UniqueName: \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\") pod \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\" (UID: \"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5\") " Oct 31 03:11:27.352349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.352326 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca" pod "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" (UID: "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:27.353368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.353331 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:27.354276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.354255 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca" pod "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" (UID: "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:27.354908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.354884 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg" pod "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" (UID: "98f6ccbb-febc-4cfb-95c5-3c5169aa65f5"). InnerVolumeSpecName "kube-api-access-vg7gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:27.455223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.455186 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7gg\" (UniqueName: \"kubernetes.io/projected/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-kube-api-access-vg7gg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:27.455223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.455223 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:27.538921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:27.538882 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:28.107782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.107751 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" Oct 31 03:11:28.107965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.107778 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt" event={"ID":"98f6ccbb-febc-4cfb-95c5-3c5169aa65f5","Type":"ContainerDied","Data":"85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad"} Oct 31 03:11:28.107965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.107824 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85efe73f5f6452c21cb462b67257d10381b8221c5256d77c815f81fabd4a32ad" Oct 31 03:11:28.267635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.267600 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v"] Oct 31 03:11:28.268502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268478 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6c199759-0aca-46cf-a4b6-38f5c32ec4d1" containerName="revoke-keys" Oct 31 03:11:28.268502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268502 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c199759-0aca-46cf-a4b6-38f5c32ec4d1" containerName="revoke-keys" Oct 31 03:11:28.268665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268525 4556 cpu_manager.go:414] "containerMap: removing container" podUID="98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" containerName="revoke-keys" Oct 31 03:11:28.268665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268533 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" containerName="revoke-keys" Oct 31 03:11:28.268776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268732 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c199759-0aca-46cf-a4b6-38f5c32ec4d1" containerName="revoke-keys" Oct 31 03:11:28.268776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.268751 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" containerName="revoke-keys" Oct 31 03:11:28.272916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.272891 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.278249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.278221 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:11:28.278389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.278313 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:11:28.283936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.283901 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v"] Oct 31 03:11:28.315800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.315719 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:28.368356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.368252 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsj8\" (UniqueName: \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.368647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.368618 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.368877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.368856 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.416508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.416469 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:11:28.422308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.422274 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.434165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.434131 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:11:28.470192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.470154 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.470437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.470269 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsj8\" (UniqueName: \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.470537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.470517 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.471341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.471311 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.472798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.472777 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.480994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.480965 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:11:28.483424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.483269 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsj8\" (UniqueName: \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\") pod \"maas-api-revoke-sub-ab16c1e0-qrn6v\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.486029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.486007 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.506583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.506534 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:11:28.516758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.515406 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:11:28.572245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572192 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572267 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572402 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572530 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572617 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572663 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdl8\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.572835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572813 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.573008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.572989 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.573068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.573034 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.615946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.615909 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674356 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674485 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674528 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674572 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674597 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674635 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv7n\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674664 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674690 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674738 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674776 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674804 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674824 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674868 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674899 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674918 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdl8\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674952 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.674981 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.675099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.675038 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.676104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.675771 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.676104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.675825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.684102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.677336 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.684102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.677848 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.684102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.678071 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.684102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.680145 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.684527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.684481 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.691896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.691830 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdl8\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.695740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.695700 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.741208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.740970 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:28.776789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.776741 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777210 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777292 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777430 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777675 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777776 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.777966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777895 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7n\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.777964 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.778031 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.778124 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.778162 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.778189 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.778783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.778757 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.779187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.779161 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.781592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.781535 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.782176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.782153 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.790112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.790080 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.790288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.790196 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv7n\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.829516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.829477 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:28.906916 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:28.906880 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0c6e7a_d3d2_4e29_8704_049cdf360eb6.slice/crio-c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de WatchSource:0}: Error finding container c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de: Status 404 returned error can't find the container with id c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de Oct 31 03:11:28.920145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:28.919827 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v"] Oct 31 03:11:29.046452 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:29.046397 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba3bee4_df5b_4a7c_9c0e_b9a5aa40a323.slice/crio-2f222f9c8cc94dd717afa435759e597f638e2ce08522dbe59d7a4d34487aa573 WatchSource:0}: Error finding container 2f222f9c8cc94dd717afa435759e597f638e2ce08522dbe59d7a4d34487aa573: Status 404 returned error can't find the container with id 2f222f9c8cc94dd717afa435759e597f638e2ce08522dbe59d7a4d34487aa573 Oct 31 03:11:29.046574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.046531 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:11:29.049378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.049336 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.049505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.049480 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.049564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.049532 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.133118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.133085 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" event={"ID":"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323","Type":"ContainerStarted","Data":"2f222f9c8cc94dd717afa435759e597f638e2ce08522dbe59d7a4d34487aa573"} Oct 31 03:11:29.135233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.135196 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" event={"ID":"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6","Type":"ContainerStarted","Data":"c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de"} Oct 31 03:11:29.136008 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:29.135982 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a34a51_f8e3_4da6_98d5_6638b944102c.slice/crio-cf3e982d12071cca5b05a21c2bc572727220d040fbb1cce574e06377ad816956 WatchSource:0}: Error finding container cf3e982d12071cca5b05a21c2bc572727220d040fbb1cce574e06377ad816956: Status 404 returned error can't find the container with id cf3e982d12071cca5b05a21c2bc572727220d040fbb1cce574e06377ad816956 Oct 31 03:11:29.137662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.137635 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:11:29.139041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.139004 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.139134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.139092 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.139194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.139140 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:29.277713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.277598 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:29.710465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.710390 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6971-87c502"] Oct 31 03:11:29.733034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.733007 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:29.745488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.741300 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bshq7\"" Oct 31 03:11:29.797224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.797145 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-87c502"] Oct 31 03:11:29.826365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.826317 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrlk\" (UniqueName: \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\") pod \"test-curl-6971-87c502\" (UID: \"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39\") " pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:29.964506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:29.955498 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrlk\" (UniqueName: \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\") pod \"test-curl-6971-87c502\" (UID: \"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39\") " pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:30.241000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.240889 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" event={"ID":"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6","Type":"ContainerStarted","Data":"f30acbeda4ec4fe49d7c8e5047dba912818f478692781ac75ea4039be506e013"} Oct 31 03:11:30.395258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.359362 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrlk\" (UniqueName: \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\") pod \"test-curl-6971-87c502\" (UID: \"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39\") " pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:30.403335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.403154 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:30.508631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.508482 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" podStartSLOduration=2.508438932 podStartE2EDuration="2.508438932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:30.413010269 +0000 UTC m=+968.268027410" watchObservedRunningTime="2026-10-31 03:11:30.508438932 +0000 UTC m=+968.363456076" Oct 31 03:11:30.548440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.516887 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" event={"ID":"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323","Type":"ContainerStarted","Data":"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997"} Oct 31 03:11:30.584448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.584389 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" event={"ID":"01a34a51-f8e3-4da6-98d5-6638b944102c","Type":"ContainerStarted","Data":"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e"} Oct 31 03:11:30.584685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.584666 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" event={"ID":"01a34a51-f8e3-4da6-98d5-6638b944102c","Type":"ContainerStarted","Data":"cf3e982d12071cca5b05a21c2bc572727220d040fbb1cce574e06377ad816956"} Oct 31 03:11:30.584810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.551271 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podStartSLOduration=2.551252078 podStartE2EDuration="2.551252078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:30.510148067 +0000 UTC m=+968.365165212" watchObservedRunningTime="2026-10-31 03:11:30.551252078 +0000 UTC m=+968.406269220" Oct 31 03:11:30.668396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.668351 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:11:30.679669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.679604 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podStartSLOduration=2.679582134 podStartE2EDuration="2.679582134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:30.602123403 +0000 UTC m=+968.457140557" watchObservedRunningTime="2026-10-31 03:11:30.679582134 +0000 UTC m=+968.534599276" Oct 31 03:11:30.716230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.716182 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.754478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.744680 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:30.756075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.756040 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.756336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.756309 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:30.789352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.788512 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.848457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.847499 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:30.848659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.848558 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:30.848659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.848614 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:30.928729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.900544 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:11:30.973735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.972525 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:30.993017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:30.990733 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.027575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.027526 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:31.058924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.058820 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.115617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.115566 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.141690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.141641 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.154566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.154515 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:11:31.197918 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:31.197875 4556 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0c6e7a_d3d2_4e29_8704_049cdf360eb6.slice/crio-f30acbeda4ec4fe49d7c8e5047dba912818f478692781ac75ea4039be506e013.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:11:31.245556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.244504 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:11:31.304747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.303508 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:31.561791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.554871 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:31.646464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.624654 4556 generic.go:356] "Generic (PLEG): container finished" podID="3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" containerID="f30acbeda4ec4fe49d7c8e5047dba912818f478692781ac75ea4039be506e013" exitCode=0 Oct 31 03:11:31.646464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.626298 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" event={"ID":"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6","Type":"ContainerDied","Data":"f30acbeda4ec4fe49d7c8e5047dba912818f478692781ac75ea4039be506e013"} Oct 31 03:11:31.786440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.782812 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:11:32.939002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.881291 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:32.939643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.891022 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:32.939869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.939821 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:32.940017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.049598 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:32.940511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.062134 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:32.957800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.089104 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.958137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.092450 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:32.958286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.156321 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.958447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.266603 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:11:32.958587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.307045 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:11:32.958733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.310620 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.966603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.316911 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:32.966858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.568608 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.967038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.846516 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.996589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.003971 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:32.996879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:31.947823 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:32.997098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.997054 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:32.997230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.183268 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:32.997405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:32.997361 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:33.812501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.766077 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.812501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.766141 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:33.812501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.799514 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded" Oct 31 03:11:33.812501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.801370 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:33.812501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.801680 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:33.877439 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:33.873700 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.556s" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007164 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007233 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007272 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007305 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007341 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007397 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:34.007442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007450 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007485 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007517 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007551 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007587 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007628 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.007663 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:33.967901 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.008252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.008070 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:34.021590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.021545 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.045143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.033521 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:34.797458 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:34.709327 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefa8dd5a_c37d_4ad8_b1ba_50e854f68b39.slice/crio-4eed53cf8314d645dbf4a51f31233924a4a6d929fc522a868d35c470245fc3f5 WatchSource:0}: Error finding container 4eed53cf8314d645dbf4a51f31233924a4a6d929fc522a868d35c470245fc3f5: Status 404 returned error can't find the container with id 4eed53cf8314d645dbf4a51f31233924a4a6d929fc522a868d35c470245fc3f5 Oct 31 03:11:34.797458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.776584 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.797458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.776647 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:34.797458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.779351 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:11:34.797458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.780698 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:34.801543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.801497 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:34.805030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803394 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:34.809530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803448 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:34.818488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803471 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:11:34.819889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803491 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.866987 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803531 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803551 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803583 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803603 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803621 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.843105 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.894982 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.896024 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:11:34.934498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:34.803510 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:11:35.149502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.119257 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 31 03:11:35.260640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.237537 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:35.482498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.482355 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6971-87c502"] Oct 31 03:11:35.550440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.517239 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-87c502" event={"ID":"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39","Type":"ContainerStarted","Data":"4eed53cf8314d645dbf4a51f31233924a4a6d929fc522a868d35c470245fc3f5"} Oct 31 03:11:35.760495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.759338 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:35.760495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.759400 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:35.818487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.818020 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.818702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.818498 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.830833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.826525 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.850305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.849555 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.850305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.849862 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:35.850305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.850099 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:35.876438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.850757 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:35.878184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.878138 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:35.886812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.851966 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82"] Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.896528 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.897130 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.897478 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.897758 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.898008 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:35.912492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:35.898240 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:36.027520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.018711 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.067470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.046925 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:36.106611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.076909 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:36.173813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.173779 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.211113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.211081 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82"] Oct 31 03:11:36.291553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.267197 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.291553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.267351 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhrh\" (UniqueName: \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.291553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.267534 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.374734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.374685 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.375161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.375138 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhrh\" (UniqueName: \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.420626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.401045 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:36.935645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.923777 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.015772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:36.987072 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.015772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.014619 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:37.084501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.070535 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.112719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.112674 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.113175 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.113431 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.113708 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.379632 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.135692 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.132266 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.139559 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.146665 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.147846 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.147649 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:11:37.444185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.262157 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.492481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.455837 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.492481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.455918 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.492481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.485944 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.492481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.486026 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.521436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.521373 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.525156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.525115 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.525475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.525434 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:37.604268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.604162 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.604603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.604566 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.640471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.640397 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.640742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.640714 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:37.696524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.696480 4556 patch_prober.go:29] interesting pod/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.710447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.706124 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.712439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.712394 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhrh\" (UniqueName: \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:37.818605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.738488 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:37.856715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.767817 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:37.926662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.926360 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:37.935546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.788245 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.936002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.935967 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:37.936404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.935238 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:37.949350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.929447 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:37.981784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:37.981728 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:37.982061 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:37.972021 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.706s" Oct 31 03:11:38.173439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.152385 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.173439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.162681 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.173439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.163051 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:11:38.173439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.163357 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:38.214505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.192284 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:38.447714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.307549 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.450930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.330620 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:38.451207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.451160 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.451385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.355947 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.451938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.338765 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5b94638-tkt82\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:38.452104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.364841 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:11:38.461578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.364801 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.475436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.400163 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:11:38.475436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.416004 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:38.475712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.416034 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:38.475968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.415948 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.502454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.416069 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:11:38.532701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.532656 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:38.581442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.577652 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:38.700496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.690528 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:38.723492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.723188 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:39.009147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:38.982753 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.055711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.045589 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 31 03:11:39.092497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.083737 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" event={"ID":"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6","Type":"ContainerDied","Data":"c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de"} Oct 31 03:11:39.092497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.083807 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30e42e1e5a2921d7a1cea9e150665de2690c2c89bf2b9fed3a1c7221e6594de" Oct 31 03:11:39.092497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.056406 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\") pod \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " Oct 31 03:11:39.092497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.084058 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\" (UniqueName: \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\") pod \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " Oct 31 03:11:39.092497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.084330 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\") pod \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\" (UID: \"3f0c6e7a-d3d2-4e29-8704-049cdf360eb6\") " Oct 31 03:11:39.107500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.106934 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.107500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.106999 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:39.171648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.171559 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.199405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.199361 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:11:39.213152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.213098 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:11:39.328163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.276106 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.331717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.331568 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:39.342708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.312124 4556 patch_prober.go:29] interesting pod/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:39.346441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.342915 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.346696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.315067 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca" pod "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" (UID: "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:39.366441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.349770 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:39.685514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.413233 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:11:39.686498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.520324 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.712218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.520367 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:11:39.717444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.520386 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:39.737441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.521005 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.737985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.567563 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.755440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.597901 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.757234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.650076 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:39.757783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.650115 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:39.758233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.650139 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:11:39.758609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.650160 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:39.758984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.672129 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca" pod "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" (UID: "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:39.802040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.801998 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.802337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.802303 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:39.803326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.803298 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:39.804102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.804071 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8" pod "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" (UID: "3f0c6e7a-d3d2-4e29-8704-049cdf360eb6"). InnerVolumeSpecName "kube-api-access-6jsj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:39.842387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.841533 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:39.842387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.841603 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:39.917472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:39.906460 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jsj8\" (UniqueName: \"kubernetes.io/projected/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6-kube-api-access-6jsj8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:40.036438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.010170 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:11:40.121438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.104522 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 31 03:11:40.175101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.175009 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v" Oct 31 03:11:40.201896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.201046 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.298382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.298337 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.711531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.711488 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.725866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.717204 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.748619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.748568 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:11:40.749685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.749653 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:11:40.750208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.750182 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.775088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.774985 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.776684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.776652 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:11:40.777065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.777029 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:40.777382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.777360 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:11:40.923443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.833135 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.060136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.054463 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:41.060136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.054151 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:11:41.060136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.054632 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:11:41.060136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:40.901557 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:41.153448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.144698 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.189959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.169575 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:41.189959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.155467 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:11:41.189959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.169732 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:11:41.189959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.180938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:41.207465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.205779 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:11:41.207465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.205842 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:11:41.263713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.262462 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:11:41.263713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.262522 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:11:41.290504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.265969 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:11:41.290504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.266030 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:11:41.290504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.268163 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:11:41.426293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.426262 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:11:41.437742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.437713 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:11:41.447653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.447605 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" event={"ID":"1d7de9b4-3d34-48b1-adf2-df8dd25893cb","Type":"ContainerStarted","Data":"32e18cbe1845618ac9f5d908dfef9b2fd59dbf2069d6f0ca2cfc051aeb1c55fa"} Oct 31 03:11:41.560682 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.560588 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw"] Oct 31 03:11:41.562028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.562003 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" containerName="revoke-keys" Oct 31 03:11:41.562213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.562195 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" containerName="revoke-keys" Oct 31 03:11:41.562585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.562568 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" containerName="revoke-keys" Oct 31 03:11:41.631268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.631234 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw"] Oct 31 03:11:41.631631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.631549 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.650329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.650299 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82"] Oct 31 03:11:41.732571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.732534 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.733140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.732679 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.733140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.732846 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftn5q\" (UniqueName: \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.744232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.742838 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.744232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.743578 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:41.838557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.838511 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftn5q\" (UniqueName: \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.838746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.838686 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.839109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.838812 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.839900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.839821 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.841472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.841150 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:41.841472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.841191 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:41.851598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.850856 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:41.856051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:41.856024 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftn5q\" (UniqueName: \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\") pod \"maas-api-revoke-sub-79d86f9b-xhmmw\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:42.022084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.021770 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:42.239566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.238701 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8"] Oct 31 03:11:42.249457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.246502 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8"] Oct 31 03:11:42.249457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.246624 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.350065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.350021 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26cqp\" (UniqueName: \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.350301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.350279 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.350480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.350460 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.454549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.454378 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.454719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.454607 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.454859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.454840 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26cqp\" (UniqueName: \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.458030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.457998 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.467535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.462746 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.520511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.504254 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26cqp\" (UniqueName: \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\") pod \"maas-api-revoke-sub-3a0f201f-dksh8\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:42.673095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.670796 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:42.758391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.742746 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.758391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.755537 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:42.762599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.762563 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" event={"ID":"1d7de9b4-3d34-48b1-adf2-df8dd25893cb","Type":"ContainerStarted","Data":"9b932d92cf8c04bb6edda261388fade156aaaf607ef8398197fee3443fd423ed"} Oct 31 03:11:42.762867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.762818 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw"] Oct 31 03:11:42.766026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.765991 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" event={"ID":"b830dbac-f3f6-43e3-9293-194fadc7dfd3","Type":"ContainerStarted","Data":"9b6120473d041d4dd3b0a09917e7bb3144ba8a8122e2e3ef0236c6070402fc39"} Oct 31 03:11:42.837514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.832073 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:42.838265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.838216 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:42.844824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.844759 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" podStartSLOduration=9.844738904 podStartE2EDuration="9.844738904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:42.811399696 +0000 UTC m=+980.666416838" watchObservedRunningTime="2026-10-31 03:11:42.844738904 +0000 UTC m=+980.699756046" Oct 31 03:11:42.866563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:42.866532 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:43.339981 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:43.339943 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ead49da_7d35_410e_889f_c17ea8187468.slice/crio-a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37 WatchSource:0}: Error finding container a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37: Status 404 returned error can't find the container with id a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37 Oct 31 03:11:43.343196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.343166 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8"] Oct 31 03:11:43.450764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.450731 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:11:43.461587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.459708 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.465641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.465615 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:11:43.558832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.558789 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.558969 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559003 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559088 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559247 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559290 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6ks\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559308 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559357 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.559468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.559377 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.660557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.660404 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.660557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.660513 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.660794 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.660883 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6ks\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.660933 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.661069 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.661111 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.661187 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.661350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.661290 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.662588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.662553 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.662857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.662819 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.662895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.662825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.663006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.662978 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.663153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.663132 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.667098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.667066 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.695651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.695619 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.741323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.741166 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:43.742313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.742284 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.742595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.742571 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:43.803490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.803004 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.823493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.820250 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6ks\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\") pod \"e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.833922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.831810 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:43.833922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.831875 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:43.833922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.832210 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:43.835101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.835074 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:43.854710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.854655 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" event={"ID":"6ead49da-7d35-410e-889f-c17ea8187468","Type":"ContainerStarted","Data":"a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37"} Oct 31 03:11:43.858477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.858373 4556 generic.go:356] "Generic (PLEG): container finished" podID="1d7de9b4-3d34-48b1-adf2-df8dd25893cb" containerID="9b932d92cf8c04bb6edda261388fade156aaaf607ef8398197fee3443fd423ed" exitCode=0 Oct 31 03:11:43.858953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.858877 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" event={"ID":"1d7de9b4-3d34-48b1-adf2-df8dd25893cb","Type":"ContainerDied","Data":"9b932d92cf8c04bb6edda261388fade156aaaf607ef8398197fee3443fd423ed"} Oct 31 03:11:43.861865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.861794 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" event={"ID":"b830dbac-f3f6-43e3-9293-194fadc7dfd3","Type":"ContainerStarted","Data":"6bb0569d56638a76cdc7d3aaf1318cfcef4c369550dd7fc6b4ab367a47ed8ffe"} Oct 31 03:11:43.953202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:43.952926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:44.131778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.131659 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:44.201440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.167941 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" podStartSLOduration=7.167919645 podStartE2EDuration="7.167919645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:43.967652706 +0000 UTC m=+981.822669872" watchObservedRunningTime="2026-10-31 03:11:44.167919645 +0000 UTC m=+982.022936787" Oct 31 03:11:44.295751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.295097 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:44.519510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.519469 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:11:44.527646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.527599 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:44.527796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.527710 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:44.527796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.527759 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:44.743779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.742839 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.743779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.742897 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 03:11:44.820436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.816255 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:44.860369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.860011 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:11:44.860369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.860074 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 03:11:44.973488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.939289 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" event={"ID":"6ead49da-7d35-410e-889f-c17ea8187468","Type":"ContainerStarted","Data":"cefa458e55338a559db5d1219ba7343351ab3a32cce74f2a932e712135fe1b84"} Oct 31 03:11:44.973488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.943390 4556 generic.go:356] "Generic (PLEG): container finished" podID="b830dbac-f3f6-43e3-9293-194fadc7dfd3" containerID="6bb0569d56638a76cdc7d3aaf1318cfcef4c369550dd7fc6b4ab367a47ed8ffe" exitCode=0 Oct 31 03:11:44.973488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.943602 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" event={"ID":"b830dbac-f3f6-43e3-9293-194fadc7dfd3","Type":"ContainerDied","Data":"6bb0569d56638a76cdc7d3aaf1318cfcef4c369550dd7fc6b4ab367a47ed8ffe"} Oct 31 03:11:44.992441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.978404 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" event={"ID":"8fcd2182-c5c4-417d-b08b-4afee41ed04f","Type":"ContainerStarted","Data":"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff"} Oct 31 03:11:44.992441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.978484 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" event={"ID":"8fcd2182-c5c4-417d-b08b-4afee41ed04f","Type":"ContainerStarted","Data":"5b17666ab27ec5a1c733047d76e58e35ac2726e5de3dc15bcf9f572a4011792d"} Oct 31 03:11:45.005448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:44.999525 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:45.176748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.176672 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" podStartSLOduration=3.176650928 podStartE2EDuration="3.176650928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:45.055962086 +0000 UTC m=+982.910979230" watchObservedRunningTime="2026-10-31 03:11:45.176650928 +0000 UTC m=+983.031668070" Oct 31 03:11:45.207055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.206976 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podStartSLOduration=2.20695977 podStartE2EDuration="2.20695977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:45.154232744 +0000 UTC m=+983.009249909" watchObservedRunningTime="2026-10-31 03:11:45.20695977 +0000 UTC m=+983.061976912" Oct 31 03:11:45.607436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.602911 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:45.656850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.656743 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:45.817258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.797987 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\" (UniqueName: \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\") pod \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " Oct 31 03:11:45.817258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.798166 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\") pod \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " Oct 31 03:11:45.817258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.798271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\") pod \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\" (UID: \"1d7de9b4-3d34-48b1-adf2-df8dd25893cb\") " Oct 31 03:11:45.848935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.817558 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca" pod "1d7de9b4-3d34-48b1-adf2-df8dd25893cb" (UID: "1d7de9b4-3d34-48b1-adf2-df8dd25893cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:45.848935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.848090 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:45.850979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.849770 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:45.850979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.849844 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:45.851191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.851018 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh" pod "1d7de9b4-3d34-48b1-adf2-df8dd25893cb" (UID: "1d7de9b4-3d34-48b1-adf2-df8dd25893cb"). InnerVolumeSpecName "kube-api-access-xhhrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:45.904578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.904541 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhhrh\" (UniqueName: \"kubernetes.io/projected/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-kube-api-access-xhhrh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:45.905088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.905068 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:45.952781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:45.952703 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca" pod "1d7de9b4-3d34-48b1-adf2-df8dd25893cb" (UID: "1d7de9b4-3d34-48b1-adf2-df8dd25893cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:46.022441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.016910 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de9b4-3d34-48b1-adf2-df8dd25893cb-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:46.071505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.067600 4556 generic.go:356] "Generic (PLEG): container finished" podID="6ead49da-7d35-410e-889f-c17ea8187468" containerID="cefa458e55338a559db5d1219ba7343351ab3a32cce74f2a932e712135fe1b84" exitCode=0 Oct 31 03:11:46.071505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.067792 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" event={"ID":"6ead49da-7d35-410e-889f-c17ea8187468","Type":"ContainerDied","Data":"cefa458e55338a559db5d1219ba7343351ab3a32cce74f2a932e712135fe1b84"} Oct 31 03:11:46.121961 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:46.121149 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:11:46.146452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.138670 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" Oct 31 03:11:46.146452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.143385 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:46.146452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.143496 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82" event={"ID":"1d7de9b4-3d34-48b1-adf2-df8dd25893cb","Type":"ContainerDied","Data":"32e18cbe1845618ac9f5d908dfef9b2fd59dbf2069d6f0ca2cfc051aeb1c55fa"} Oct 31 03:11:46.146452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.143908 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e18cbe1845618ac9f5d908dfef9b2fd59dbf2069d6f0ca2cfc051aeb1c55fa" Oct 31 03:11:46.363498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.359016 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:46.480438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.457704 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:11:46.524046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.494523 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1d7de9b4-3d34-48b1-adf2-df8dd25893cb" containerName="revoke-keys" Oct 31 03:11:46.524046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.494552 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7de9b4-3d34-48b1-adf2-df8dd25893cb" containerName="revoke-keys" Oct 31 03:11:46.524046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.494785 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7de9b4-3d34-48b1-adf2-df8dd25893cb" containerName="revoke-keys" Oct 31 03:11:46.533437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.530953 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.559225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.556892 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8a8b17ff\"" Oct 31 03:11:46.559675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.559638 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8a8b17ff-dockercfg-242m4\"" Oct 31 03:11:46.563603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.563489 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:11:46.602372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.598316 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:11:46.678510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.620096 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.679552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.679501 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbbz\" (UniqueName: \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.679845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.679821 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.768490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.755579 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:11:46.811263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.811224 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbbz\" (UniqueName: \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.811632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.811603 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.811890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.811866 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jb5h\" (UniqueName: \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.813507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.813482 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.813997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.813342 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.858436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.857574 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:46.858436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.858334 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:46.858436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.858388 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:46.912654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.887637 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:46.912654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.887723 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:46.912654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.911959 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:46.924531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.921902 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:46.924531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.921966 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:46.935886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.928724 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jb5h\" (UniqueName: \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.935886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.928775 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.935886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.931174 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:46.939630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.939477 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbbz\" (UniqueName: \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\") pod \"payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:46.940268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:46.940223 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:47.051692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.051654 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jb5h\" (UniqueName: \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\") pod \"payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:47.109541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.109506 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:47.145242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.145109 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:47.165656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.145654 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:47.276022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.275781 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:47.387440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.367585 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\") pod \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " Oct 31 03:11:47.387440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.367733 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\") pod \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " Oct 31 03:11:47.387440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.368006 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\" (UniqueName: \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\") pod \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\" (UID: \"b830dbac-f3f6-43e3-9293-194fadc7dfd3\") " Oct 31 03:11:47.454479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.453510 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca" pod "b830dbac-f3f6-43e3-9293-194fadc7dfd3" (UID: "b830dbac-f3f6-43e3-9293-194fadc7dfd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:47.509730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.495576 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:47.544658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.544580 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca" pod "b830dbac-f3f6-43e3-9293-194fadc7dfd3" (UID: "b830dbac-f3f6-43e3-9293-194fadc7dfd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:47.777106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.746515 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b830dbac-f3f6-43e3-9293-194fadc7dfd3-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:47.840228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.839584 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q" pod "b830dbac-f3f6-43e3-9293-194fadc7dfd3" (UID: "b830dbac-f3f6-43e3-9293-194fadc7dfd3"). InnerVolumeSpecName "kube-api-access-ftn5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:47.889817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.841666 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:47.889817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.841722 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:47.889817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.853696 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftn5q\" (UniqueName: \"kubernetes.io/projected/b830dbac-f3f6-43e3-9293-194fadc7dfd3-kube-api-access-ftn5q\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:47.889817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.862950 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:47.960514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.955740 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:47.960514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:47.955824 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:48.017975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.017284 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:48.017975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.017352 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:48.040128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.039954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:48.653434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.638843 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:48.894669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.867380 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:48.909497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.878388 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:48.917679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.912879 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:48.935203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.878519 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:48.935322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.931035 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" Oct 31 03:11:49.005024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:48.989599 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:49.013407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.013364 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw" event={"ID":"b830dbac-f3f6-43e3-9293-194fadc7dfd3","Type":"ContainerDied","Data":"9b6120473d041d4dd3b0a09917e7bb3144ba8a8122e2e3ef0236c6070402fc39"} Oct 31 03:11:49.024364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.024336 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6120473d041d4dd3b0a09917e7bb3144ba8a8122e2e3ef0236c6070402fc39" Oct 31 03:11:49.030514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.028144 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" event={"ID":"6ead49da-7d35-410e-889f-c17ea8187468","Type":"ContainerDied","Data":"a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37"} Oct 31 03:11:49.041474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.039566 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59706c8042c2c3ad6059b6c19761b320e7c28f180474e96ef6ceb2bb3793e37" Oct 31 03:11:49.041474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.031877 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8" Oct 31 03:11:49.059800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.059772 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:49.060025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.060002 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:49.099296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.099244 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\") pod \"6ead49da-7d35-410e-889f-c17ea8187468\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " Oct 31 03:11:49.102568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.100407 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\" (UniqueName: \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\") pod \"6ead49da-7d35-410e-889f-c17ea8187468\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " Oct 31 03:11:49.102568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.101162 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:11:49.102568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.101213 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.138188 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca" pod "6ead49da-7d35-410e-889f-c17ea8187468" (UID: "6ead49da-7d35-410e-889f-c17ea8187468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.138285 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\") pod \"6ead49da-7d35-410e-889f-c17ea8187468\" (UID: \"6ead49da-7d35-410e-889f-c17ea8187468\") " Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.139116 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.142527 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.142622 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:49.143739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.142677 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca" pod "6ead49da-7d35-410e-889f-c17ea8187468" (UID: "6ead49da-7d35-410e-889f-c17ea8187468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:49.144946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.144865 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:11:49.166462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.166427 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp" pod "6ead49da-7d35-410e-889f-c17ea8187468" (UID: "6ead49da-7d35-410e-889f-c17ea8187468"). InnerVolumeSpecName "kube-api-access-26cqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:49.198726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.190495 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:11:49.288544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.264120 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26cqp\" (UniqueName: \"kubernetes.io/projected/6ead49da-7d35-410e-889f-c17ea8187468-kube-api-access-26cqp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:49.288874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.288846 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ead49da-7d35-410e-889f-c17ea8187468-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:49.418166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.417220 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:49.714787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.714260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:49.738459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.738435 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:11:49.806225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.804636 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:49.807511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.807257 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:11:49.836737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.836640 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:49.836737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:49.836690 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:50.044799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.044760 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" event={"ID":"b241f5f0-cfdd-4106-9589-070412908259","Type":"ContainerStarted","Data":"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7"} Oct 31 03:11:50.045283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.044891 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" event={"ID":"b241f5f0-cfdd-4106-9589-070412908259","Type":"ContainerStarted","Data":"d286c91e6aa6b75586635e003d1483733830a575fccca1d5a83827c76816bc3b"} Oct 31 03:11:50.045283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.045022 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:11:50.047933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.047902 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" event={"ID":"47c9ab2f-971a-4f63-b746-a068e5a3e702","Type":"ContainerStarted","Data":"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8"} Oct 31 03:11:50.048098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.047946 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" event={"ID":"47c9ab2f-971a-4f63-b746-a068e5a3e702","Type":"ContainerStarted","Data":"3be3a4232d74e51c662b20f6f8a86b96c77bceddc09b0417ff70b328c4ca10d1"} Oct 31 03:11:50.093435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.068070 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" podStartSLOduration=4.068050764 podStartE2EDuration="4.068050764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:50.065929522 +0000 UTC m=+987.920946664" watchObservedRunningTime="2026-10-31 03:11:50.068050764 +0000 UTC m=+987.923067907" Oct 31 03:11:50.104758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.104698 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" podStartSLOduration=4.104679899 podStartE2EDuration="4.104679899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:50.095019138 +0000 UTC m=+987.950036280" watchObservedRunningTime="2026-10-31 03:11:50.104679899 +0000 UTC m=+987.959697043" Oct 31 03:11:50.556057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.556023 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:50.705474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.702788 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:50.836107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.836034 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:50.836256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.836099 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:50.842288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:50.842257 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:51.010090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.010053 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:11:51.010523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.010493 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:51.011387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.011367 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6ead49da-7d35-410e-889f-c17ea8187468" containerName="revoke-keys" Oct 31 03:11:51.011551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.011537 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ead49da-7d35-410e-889f-c17ea8187468" containerName="revoke-keys" Oct 31 03:11:51.011677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.011668 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b830dbac-f3f6-43e3-9293-194fadc7dfd3" containerName="revoke-keys" Oct 31 03:11:51.011758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.011747 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b830dbac-f3f6-43e3-9293-194fadc7dfd3" containerName="revoke-keys" Oct 31 03:11:51.012013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.012000 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ead49da-7d35-410e-889f-c17ea8187468" containerName="revoke-keys" Oct 31 03:11:51.012466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.012090 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b830dbac-f3f6-43e3-9293-194fadc7dfd3" containerName="revoke-keys" Oct 31 03:11:51.024437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.024160 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.032629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.032598 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:11:51.042109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.039724 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1adda1ee-openshift-default-dockercfg-8hj58\"" Oct 31 03:11:51.065130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.065104 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:11:51.066963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.066935 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.067122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.067106 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.067267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.067250 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.067367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.067349 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.078439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.071540 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.093516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.078765 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.093852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.093823 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.098698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.098666 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.098997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.098976 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brts7\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.192444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.189895 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228512 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228575 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228676 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228817 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brts7\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228949 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.228983 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.229435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.229077 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.259082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.238905 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.259082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.238984 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.259082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.239124 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.259082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.239395 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.259082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.248200 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.318438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.272479 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.318438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.272748 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.379151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.379039 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.383913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.383883 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.385910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.385839 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:51.397613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.386900 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.409809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.409727 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brts7\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\") pod \"e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.660934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.660894 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:51.697998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.697902 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:51.837018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.836882 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:51.837018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.836932 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:51.987920 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:51.987880 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be131a2_d7bd_421c_9f0f_dc00722268b6.slice/crio-f015fa8e08a0cbeec4598e5b5568a0aaf2ba316dfc057810574e976a98e072f5 WatchSource:0}: Error finding container f015fa8e08a0cbeec4598e5b5568a0aaf2ba316dfc057810574e976a98e072f5: Status 404 returned error can't find the container with id f015fa8e08a0cbeec4598e5b5568a0aaf2ba316dfc057810574e976a98e072f5 Oct 31 03:11:51.990253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.990228 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:11:51.990998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.990964 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:51.991065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.991047 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:51.991105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:51.991092 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:52.043152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.034354 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:52.078739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.078696 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6971-87c502" event={"ID":"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39","Type":"ContainerStarted","Data":"088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518"} Oct 31 03:11:52.082325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.082294 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" event={"ID":"4be131a2-d7bd-421c-9f0f-dc00722268b6","Type":"ContainerStarted","Data":"f015fa8e08a0cbeec4598e5b5568a0aaf2ba316dfc057810574e976a98e072f5"} Oct 31 03:11:52.121548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.121476 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6971-87c502" podStartSLOduration=6.403965616 podStartE2EDuration="23.121457209s" totalImagesPullingTime="16.717491593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:52.095640212 +0000 UTC m=+989.950657359" watchObservedRunningTime="2026-10-31 03:11:52.121457209 +0000 UTC m=+989.976474350" Oct 31 03:11:52.434143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.434102 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t"] Oct 31 03:11:52.439503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.439269 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.443704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.443189 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:11:52.443704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.443229 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:11:52.450496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.450084 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t"] Oct 31 03:11:52.564145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.564103 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.564145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.564153 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.564479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.564266 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwjh\" (UniqueName: \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.665708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.665666 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.665708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.665712 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.665947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.665767 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwjh\" (UniqueName: \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.666376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.666341 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.668550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.668521 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.680342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.680296 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwjh\" (UniqueName: \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\") pod \"maas-api-revoke-sub-c19ba506-4896t\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.772400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.772311 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:52.836144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.836108 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:11:52.836294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.836172 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 03:11:52.881809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.881448 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6971-87c502"] Oct 31 03:11:52.886884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:52.886844 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6971-87c502"] Oct 31 03:11:53.020859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.020814 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:53.101142 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:53.101105 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594753ca_54dc_42de_9a03_14185f7e32b8.slice/crio-8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11 WatchSource:0}: Error finding container 8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11: Status 404 returned error can't find the container with id 8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11 Oct 31 03:11:53.112057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.111257 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t"] Oct 31 03:11:53.149719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.149566 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" event={"ID":"4be131a2-d7bd-421c-9f0f-dc00722268b6","Type":"ContainerStarted","Data":"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d"} Oct 31 03:11:53.187065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.186699 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" podStartSLOduration=3.186678542 podStartE2EDuration="3.186678542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:53.174968244 +0000 UTC m=+991.029985399" watchObservedRunningTime="2026-10-31 03:11:53.186678542 +0000 UTC m=+991.041695684" Oct 31 03:11:53.661674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.661639 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:53.667702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.667670 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:53.842681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.842592 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:53.843165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.843133 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:53.843884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:53.843865 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:11:54.108901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.108872 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:11:54.109647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.109618 4556 cpu_manager.go:414] "containerMap: removing container" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" containerName="test-curl-6971-87c502" Oct 31 03:11:54.109647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.109648 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" containerName="test-curl-6971-87c502" Oct 31 03:11:54.109785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.109775 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" containerName="test-curl-6971-87c502" Oct 31 03:11:54.113698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.113679 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.116751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.116730 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8a8b17ff-serving-cert\"" Oct 31 03:11:54.128526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.125150 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:11:54.167382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.167336 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" event={"ID":"594753ca-54dc-42de-9a03-14185f7e32b8","Type":"ContainerDied","Data":"19a011f09ccb392d4517f9be31190c108ff3faf0d91f1c06c4d9006be9c8cbee"} Oct 31 03:11:54.168121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.168089 4556 generic.go:356] "Generic (PLEG): container finished" podID="594753ca-54dc-42de-9a03-14185f7e32b8" containerID="19a011f09ccb392d4517f9be31190c108ff3faf0d91f1c06c4d9006be9c8cbee" exitCode=0 Oct 31 03:11:54.168472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.168447 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6971-87c502" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" containerName="test-curl-6971-87c502" containerID="cri-o://088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518" gracePeriod=2 Oct 31 03:11:54.168600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.168569 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" event={"ID":"594753ca-54dc-42de-9a03-14185f7e32b8","Type":"ContainerStarted","Data":"8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11"} Oct 31 03:11:54.169721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.169691 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:54.170972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.170951 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:11:54.200701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.200666 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.200848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.200825 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvr4p\" (UniqueName: \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.200940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.200895 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.200999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.200955 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.211971 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:54.211933 4556 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-87c502\" is forbidden: User \"system:node:ip-10-0-139-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-52.us-east-2.compute.internal' and this object" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:54.272026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.271965 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:54.302428 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.302377 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvr4p\" (UniqueName: \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.302606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.302562 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.302786 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:54.302758 4556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-serving-cert: secret "maas-api-e2e-ait-8a8b17ff-serving-cert" not found Oct 31 03:11:54.302947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.302783 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.302947 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:54.302830 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls podName:01c28b01-3711-449c-a93a-69f2254aa1ed nodeName:}" failed. No retries permitted until 2026-10-31 03:11:54.802809777 +0000 UTC m=+992.657826925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls") pod "maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" (UID: "01c28b01-3711-449c-a93a-69f2254aa1ed") : secret "maas-api-e2e-ait-8a8b17ff-serving-cert" not found Oct 31 03:11:54.303228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.303183 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.303378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.303361 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.305569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.305542 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.315948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.315924 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvr4p\" (UniqueName: \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.457162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.457063 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:54.628006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.627968 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:54.814482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.814370 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.817473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.817431 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") pod \"maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:54.829851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.829820 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:54.970779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:54.970722 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:55.036798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.036762 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:55.373031 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:55.372936 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c28b01_3711_449c_a93a_69f2254aa1ed.slice/crio-dcdd904b5c60b97c6f5d799b92af9f145929c74b945f5927f1ea96d5d33805ce WatchSource:0}: Error finding container dcdd904b5c60b97c6f5d799b92af9f145929c74b945f5927f1ea96d5d33805ce: Status 404 returned error can't find the container with id dcdd904b5c60b97c6f5d799b92af9f145929c74b945f5927f1ea96d5d33805ce Oct 31 03:11:55.375664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.375633 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:11:55.407714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.407688 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:55.535106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.535070 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\") pod \"594753ca-54dc-42de-9a03-14185f7e32b8\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " Oct 31 03:11:55.535311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.535292 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\") pod \"594753ca-54dc-42de-9a03-14185f7e32b8\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " Oct 31 03:11:55.535742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.535677 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\" (UniqueName: \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\") pod \"594753ca-54dc-42de-9a03-14185f7e32b8\" (UID: \"594753ca-54dc-42de-9a03-14185f7e32b8\") " Oct 31 03:11:55.535958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.535777 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca" pod "594753ca-54dc-42de-9a03-14185f7e32b8" (UID: "594753ca-54dc-42de-9a03-14185f7e32b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:11:55.536594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.536565 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:55.543585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.543221 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh" pod "594753ca-54dc-42de-9a03-14185f7e32b8" (UID: "594753ca-54dc-42de-9a03-14185f7e32b8"). InnerVolumeSpecName "kube-api-access-klwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:55.543585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.543572 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca" pod "594753ca-54dc-42de-9a03-14185f7e32b8" (UID: "594753ca-54dc-42de-9a03-14185f7e32b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:11:55.545116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.545085 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:55.638606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.638517 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klwjh\" (UniqueName: \"kubernetes.io/projected/594753ca-54dc-42de-9a03-14185f7e32b8-kube-api-access-klwjh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:55.638606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:55.638546 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594753ca-54dc-42de-9a03-14185f7e32b8-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:56.042455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.040305 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:56.263442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.241137 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:56.367931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.367902 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:11:56.379148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.379114 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" event={"ID":"01c28b01-3711-449c-a93a-69f2254aa1ed","Type":"ContainerStarted","Data":"03d67bdd565c2aefdd392060f3f733f2a6a10b3b578a86449211dd69aa442621"} Oct 31 03:11:56.380436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.380383 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" event={"ID":"01c28b01-3711-449c-a93a-69f2254aa1ed","Type":"ContainerStarted","Data":"dcdd904b5c60b97c6f5d799b92af9f145929c74b945f5927f1ea96d5d33805ce"} Oct 31 03:11:56.380660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.380593 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" event={"ID":"594753ca-54dc-42de-9a03-14185f7e32b8","Type":"ContainerDied","Data":"8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11"} Oct 31 03:11:56.380862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.380842 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c9d95d4c409f63f78cdf5c64242badee887c5b8b25db1b1a58486ae7aa67f11" Oct 31 03:11:56.380948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.380464 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t" Oct 31 03:11:56.403460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.401000 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" podStartSLOduration=1.866150122 podStartE2EDuration="2.400978386s" totalImagesPullingTime="534.828264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:56.395613834 +0000 UTC m=+994.250630976" watchObservedRunningTime="2026-10-31 03:11:56.400978386 +0000 UTC m=+994.255995529" Oct 31 03:11:56.597811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.597778 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:56.645223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.645184 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\" (UniqueName: \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\") pod \"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39\" (UID: \"efa8dd5a-c37d-4ad8-b1ba-50e854f68b39\") " Oct 31 03:11:56.647591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.647563 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk" pod "efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" (UID: "efa8dd5a-c37d-4ad8-b1ba-50e854f68b39"). InnerVolumeSpecName "kube-api-access-vtrlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:11:56.685315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.685274 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:56.747289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:56.747209 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrlk\" (UniqueName: \"kubernetes.io/projected/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39-kube-api-access-vtrlk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:11:57.394568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.394529 4556 generic.go:356] "Generic (PLEG): container finished" podID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" containerID="088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518" exitCode=137 Oct 31 03:11:57.395012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.394598 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:57.395012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.394653 4556 scope.go:122] "RemoveContainer" containerID="088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518" Oct 31 03:11:57.408826 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:57.408785 4556 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6971-87c502\" is forbidden: User \"system:node:ip-10-0-139-52.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-52.us-east-2.compute.internal' and this object" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" pod="opendatahub/test-curl-6971-87c502" Oct 31 03:11:57.409809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.409789 4556 scope.go:122] "RemoveContainer" containerID="088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518" Oct 31 03:11:57.410139 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:11:57.410120 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518\": container with ID starting with 088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518 not found: ID does not exist" containerID="088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518" Oct 31 03:11:57.410202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.410149 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518"} err="failed to get container status \"088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518\": rpc error: code = NotFound desc = could not find container \"088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518\": container with ID starting with 088876c985e958c68a7029cfb1e45a729113d809ee5060b277cf6b16b4cd7518 not found: ID does not exist" Oct 31 03:11:57.445917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.445887 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:57.593587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:57.593549 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:58.270852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.270814 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa8dd5a-c37d-4ad8-b1ba-50e854f68b39" path="/var/lib/kubelet/pods/efa8dd5a-c37d-4ad8-b1ba-50e854f68b39/volumes" Oct 31 03:11:58.379225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.379190 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:58.545650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.545570 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:11:58.546296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.546280 4556 cpu_manager.go:414] "containerMap: removing container" podUID="594753ca-54dc-42de-9a03-14185f7e32b8" containerName="revoke-keys" Oct 31 03:11:58.546340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.546298 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594753ca-54dc-42de-9a03-14185f7e32b8" containerName="revoke-keys" Oct 31 03:11:58.546432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.546407 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="594753ca-54dc-42de-9a03-14185f7e32b8" containerName="revoke-keys" Oct 31 03:11:58.551906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.551884 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.563771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.562934 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:11:58.581369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.580980 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.589235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.589192 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.589389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.589249 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.589995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.589965 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590015 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590047 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7lz\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590098 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590196 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590266 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.590385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.590372 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.591346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.591294 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:11:58.604165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.604051 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:11:58.692031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.691991 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692040 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692062 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692203 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692228 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692248 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7lz\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692288 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692392 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692441 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692509 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692536 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692658 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692678 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692768 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692811 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692861 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.692921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692881 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.692957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.692921 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.693040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.693018 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.693085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.693020 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.693085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.693076 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.693143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.693131 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdbh\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.693657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.693633 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.695074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.695057 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.695125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.695106 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.702829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.702803 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch"] Oct 31 03:11:58.706977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.706955 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.707072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.707026 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7lz\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.709005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.708985 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.712586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.712233 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:11:58.712834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.712815 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:11:58.722448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.722398 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch"] Oct 31 03:11:58.794707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794675 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.794881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794730 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdbh\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.794881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794798 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.794881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794823 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.794881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794843 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.794970 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795028 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795078 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.795142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795114 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795209 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795214 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnljh\" (UniqueName: \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.795258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795242 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795323 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795371 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795474 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.795766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.795703 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.796100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.796077 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.797397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.797367 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.797856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.797839 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.805913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.805889 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.805989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.805965 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdbh\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\") pod \"e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.875689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.875654 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:58.896913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.896886 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.897196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.897174 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.897290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.897247 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnljh\" (UniqueName: \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.898032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.897984 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.899773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.899753 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:58.904879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.904771 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:11:58.909507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:58.909485 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnljh\" (UniqueName: \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\") pod \"maas-api-revoke-sub-7ce35cb0-dskch\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:59.060400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.060327 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:11:59.175892 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:59.175863 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9d0cb4_dddd_44bb_be58_a687e98c307d.slice/crio-c75cac00f252adf2e83cdba31cf3453acee67dfa25c4b2ace517f5e479bff5a4 WatchSource:0}: Error finding container c75cac00f252adf2e83cdba31cf3453acee67dfa25c4b2ace517f5e479bff5a4: Status 404 returned error can't find the container with id c75cac00f252adf2e83cdba31cf3453acee67dfa25c4b2ace517f5e479bff5a4 Oct 31 03:11:59.176485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.176458 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:11:59.179167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.179131 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.179244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.179217 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.179278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.179262 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.219862 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:11:59.219827 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf83620_6ddc_4c31_8c8c_e819239caca2.slice/crio-071602c8de5a13884912aad56d1f4de1a9b2856a5dff3c07c8d40b2b559d0274 WatchSource:0}: Error finding container 071602c8de5a13884912aad56d1f4de1a9b2856a5dff3c07c8d40b2b559d0274: Status 404 returned error can't find the container with id 071602c8de5a13884912aad56d1f4de1a9b2856a5dff3c07c8d40b2b559d0274 Oct 31 03:11:59.220626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.220604 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:11:59.222907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.222865 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.223019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.222935 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.223019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.222970 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:11:59.328094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.328059 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:59.387644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.386955 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch"] Oct 31 03:11:59.431040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.430958 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" event={"ID":"8fc17f3b-9f7f-4421-a30b-79560e81a844","Type":"ContainerStarted","Data":"599a3461022351e28359eabfefbffc20a4bbaa693e36d8c0202550d09f859e8d"} Oct 31 03:11:59.434312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.433064 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" event={"ID":"2a9d0cb4-dddd-44bb-be58-a687e98c307d","Type":"ContainerStarted","Data":"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3"} Oct 31 03:11:59.434312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.433114 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" event={"ID":"2a9d0cb4-dddd-44bb-be58-a687e98c307d","Type":"ContainerStarted","Data":"c75cac00f252adf2e83cdba31cf3453acee67dfa25c4b2ace517f5e479bff5a4"} Oct 31 03:11:59.438143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.436658 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" event={"ID":"ddf83620-6ddc-4c31-8c8c-e819239caca2","Type":"ContainerStarted","Data":"207a55dc4c745a637fa39903b46e11f2c44c36b2411a8193349ccf63dcfac048"} Oct 31 03:11:59.438143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.436707 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" event={"ID":"ddf83620-6ddc-4c31-8c8c-e819239caca2","Type":"ContainerStarted","Data":"071602c8de5a13884912aad56d1f4de1a9b2856a5dff3c07c8d40b2b559d0274"} Oct 31 03:11:59.509442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.505681 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:59.534507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.527543 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podStartSLOduration=1.527519557 podStartE2EDuration="1.527519557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:59.456454682 +0000 UTC m=+997.311471823" watchObservedRunningTime="2026-10-31 03:11:59.527519557 +0000 UTC m=+997.382536698" Oct 31 03:11:59.607444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.584467 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podStartSLOduration=1.584441566 podStartE2EDuration="1.584441566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:59.50512812 +0000 UTC m=+997.360145262" watchObservedRunningTime="2026-10-31 03:11:59.584441566 +0000 UTC m=+997.439458708" Oct 31 03:11:59.657110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.657072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:11:59.877133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.877101 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:11:59.910843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:11:59.908060 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:12:00.450664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.450627 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" event={"ID":"8fc17f3b-9f7f-4421-a30b-79560e81a844","Type":"ContainerStarted","Data":"9f5c76c265aec2a39cc8eab0038d1a9cc0c9db14f2b95c868d1b3bc59f9590f3"} Oct 31 03:12:00.484927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.484889 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:00.512435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.502036 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" podStartSLOduration=2.50201108 podStartE2EDuration="2.50201108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:00.474816279 +0000 UTC m=+998.329833422" watchObservedRunningTime="2026-10-31 03:12:00.50201108 +0000 UTC m=+998.357028222" Oct 31 03:12:00.639483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.639320 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:00.807652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.807562 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:00.878401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.878362 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:00.878738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.878707 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:00.910288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.910205 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:00.910515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.910291 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:00.950363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:00.950319 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:01.082783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.082684 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:12:01.451789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.451443 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:01.482441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.476745 4556 generic.go:356] "Generic (PLEG): container finished" podID="8fc17f3b-9f7f-4421-a30b-79560e81a844" containerID="9f5c76c265aec2a39cc8eab0038d1a9cc0c9db14f2b95c868d1b3bc59f9590f3" exitCode=0 Oct 31 03:12:01.482441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.478726 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" event={"ID":"8fc17f3b-9f7f-4421-a30b-79560e81a844","Type":"ContainerDied","Data":"9f5c76c265aec2a39cc8eab0038d1a9cc0c9db14f2b95c868d1b3bc59f9590f3"} Oct 31 03:12:01.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.880582 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:01.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.880654 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:01.909168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.906871 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:01.909168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:01.906935 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:02.116720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.116618 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:12:02.535524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.534612 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:02.842154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.838193 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p"] Oct 31 03:12:02.846401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.845885 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:02.932687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.916005 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p"] Oct 31 03:12:02.936014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.933636 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:02.936014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.933701 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:02.938534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.938490 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:02.938673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.938567 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:02.944010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.943974 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:02.944202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.944184 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:02.944264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:02.944231 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28wc\" (UniqueName: \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.021811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.021777 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:12:03.045774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.045213 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\") pod \"8fc17f3b-9f7f-4421-a30b-79560e81a844\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " Oct 31 03:12:03.045774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.045388 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\" (UniqueName: \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\") pod \"8fc17f3b-9f7f-4421-a30b-79560e81a844\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " Oct 31 03:12:03.045774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.045524 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\") pod \"8fc17f3b-9f7f-4421-a30b-79560e81a844\" (UID: \"8fc17f3b-9f7f-4421-a30b-79560e81a844\") " Oct 31 03:12:03.046056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.045878 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca" pod "8fc17f3b-9f7f-4421-a30b-79560e81a844" (UID: "8fc17f3b-9f7f-4421-a30b-79560e81a844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:03.046056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.046025 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.046242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.046218 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.046331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.046287 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28wc\" (UniqueName: \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.046587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.046561 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:03.047392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.047369 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.051178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.051154 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.051790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.051763 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca" pod "8fc17f3b-9f7f-4421-a30b-79560e81a844" (UID: "8fc17f3b-9f7f-4421-a30b-79560e81a844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:03.052236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.052202 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh" pod "8fc17f3b-9f7f-4421-a30b-79560e81a844" (UID: "8fc17f3b-9f7f-4421-a30b-79560e81a844"). InnerVolumeSpecName "kube-api-access-hnljh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:03.059736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.059678 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28wc\" (UniqueName: \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\") pod \"maas-api-revoke-sub-74c62210-pw29p\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.149221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.149091 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnljh\" (UniqueName: \"kubernetes.io/projected/8fc17f3b-9f7f-4421-a30b-79560e81a844-kube-api-access-hnljh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:03.149221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.149136 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc17f3b-9f7f-4421-a30b-79560e81a844-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:03.291018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.290700 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:03.437759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.437712 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:12:03.547468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.534016 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:03.662477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.662450 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" Oct 31 03:12:03.662477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.662469 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch" event={"ID":"8fc17f3b-9f7f-4421-a30b-79560e81a844","Type":"ContainerDied","Data":"599a3461022351e28359eabfefbffc20a4bbaa693e36d8c0202550d09f859e8d"} Oct 31 03:12:03.662641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.662521 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="599a3461022351e28359eabfefbffc20a4bbaa693e36d8c0202550d09f859e8d" Oct 31 03:12:03.938345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.919651 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:03.938345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.919723 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:03.938345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.925980 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:03.938345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.926042 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:03.938345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:03.931498 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p"] Oct 31 03:12:03.949072 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:12:03.941811 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014ee252_8356_4f00_9c7e_c10250efa491.slice/crio-3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a WatchSource:0}: Error finding container 3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a: Status 404 returned error can't find the container with id 3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a Oct 31 03:12:04.231239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.231201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:04.394439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.389592 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:04.666661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.666624 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:04.717453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.715462 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" event={"ID":"014ee252-8356-4f00-9c7e-c10250efa491","Type":"ContainerStarted","Data":"58b6402176affe264ef95dea22a48a15ccc9e3930a03836fc694ab1f414969dc"} Oct 31 03:12:04.717453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.715540 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" event={"ID":"014ee252-8356-4f00-9c7e-c10250efa491","Type":"ContainerStarted","Data":"3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a"} Oct 31 03:12:04.732784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.732360 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" podStartSLOduration=2.732340649 podStartE2EDuration="2.732340649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:04.7314439 +0000 UTC m=+1002.586461041" watchObservedRunningTime="2026-10-31 03:12:04.732340649 +0000 UTC m=+1002.587357792" Oct 31 03:12:04.809436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.808868 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:04.878462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.878408 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:04.878716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.878688 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:04.918505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.905937 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:04.918505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:04.906005 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:05.046368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.046330 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:05.439433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.439315 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:05.733089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.732051 4556 generic.go:356] "Generic (PLEG): container finished" podID="014ee252-8356-4f00-9c7e-c10250efa491" containerID="58b6402176affe264ef95dea22a48a15ccc9e3930a03836fc694ab1f414969dc" exitCode=0 Oct 31 03:12:05.733089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.732442 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" event={"ID":"014ee252-8356-4f00-9c7e-c10250efa491","Type":"ContainerDied","Data":"58b6402176affe264ef95dea22a48a15ccc9e3930a03836fc694ab1f414969dc"} Oct 31 03:12:05.900447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.887289 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:05.900447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.887368 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:05.923644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.920661 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:05.923644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:05.920723 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:06.387464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:06.384488 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:06.893168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:06.891439 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 03:12:06.893168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:06.891509 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 03:12:06.906692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:06.906538 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 03:12:06.906692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:06.906597 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 03:12:07.033228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.033201 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:07.074115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.074057 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\") pod \"014ee252-8356-4f00-9c7e-c10250efa491\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " Oct 31 03:12:07.074302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.074154 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\") pod \"014ee252-8356-4f00-9c7e-c10250efa491\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " Oct 31 03:12:07.074374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.074311 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\" (UniqueName: \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\") pod \"014ee252-8356-4f00-9c7e-c10250efa491\" (UID: \"014ee252-8356-4f00-9c7e-c10250efa491\") " Oct 31 03:12:07.074847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.074818 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca" pod "014ee252-8356-4f00-9c7e-c10250efa491" (UID: "014ee252-8356-4f00-9c7e-c10250efa491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:07.075678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.075649 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:07.090650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.083953 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca" pod "014ee252-8356-4f00-9c7e-c10250efa491" (UID: "014ee252-8356-4f00-9c7e-c10250efa491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:07.090650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.084101 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc" pod "014ee252-8356-4f00-9c7e-c10250efa491" (UID: "014ee252-8356-4f00-9c7e-c10250efa491"). InnerVolumeSpecName "kube-api-access-c28wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:07.178038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.177996 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28wc\" (UniqueName: \"kubernetes.io/projected/014ee252-8356-4f00-9c7e-c10250efa491-kube-api-access-c28wc\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:07.178038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.178041 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ee252-8356-4f00-9c7e-c10250efa491-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:07.297280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.297187 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:07.762634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.762590 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" Oct 31 03:12:07.763218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.762592 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p" event={"ID":"014ee252-8356-4f00-9c7e-c10250efa491","Type":"ContainerDied","Data":"3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a"} Oct 31 03:12:07.763374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.763236 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cecd935459dca6904bb5bc2c6383ecf39102f424e14902aa9b4a3e940bc265a" Oct 31 03:12:07.911683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.911649 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:12:07.912172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.912156 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:12:07.913073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.913052 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:12:07.924684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.924552 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:12:07.925012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.924976 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:12:07.926045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:07.926020 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:12:08.053060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:08.052977 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:08.250655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:08.250611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:09.004323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:08.998948 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml"] Oct 31 03:12:09.015502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.015475 4556 cpu_manager.go:414] "containerMap: removing container" podUID="014ee252-8356-4f00-9c7e-c10250efa491" containerName="revoke-keys" Oct 31 03:12:09.015704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.015676 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014ee252-8356-4f00-9c7e-c10250efa491" containerName="revoke-keys" Oct 31 03:12:09.020866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.020843 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8fc17f3b-9f7f-4421-a30b-79560e81a844" containerName="revoke-keys" Oct 31 03:12:09.021838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.021822 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc17f3b-9f7f-4421-a30b-79560e81a844" containerName="revoke-keys" Oct 31 03:12:09.022337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.022319 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="014ee252-8356-4f00-9c7e-c10250efa491" containerName="revoke-keys" Oct 31 03:12:09.028358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.028341 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc17f3b-9f7f-4421-a30b-79560e81a844" containerName="revoke-keys" Oct 31 03:12:09.038866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.038826 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml"] Oct 31 03:12:09.039146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.039128 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.045358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.045268 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:12:09.045358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.045331 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:12:09.127497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.127464 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rn88\" (UniqueName: \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.127736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.127705 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.127886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.127858 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.237578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.237526 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rn88\" (UniqueName: \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.263824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.263726 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.265176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.265148 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.277455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.253694 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:09.277455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.264965 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.277455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.277271 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rn88\" (UniqueName: \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.291435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.288016 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-205cd79e-bzlml\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.366475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.366444 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:09.507658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.506743 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:09.885883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:09.885564 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml"] Oct 31 03:12:11.114996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.114890 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" event={"ID":"30e269f2-8bf4-41c2-bcb3-426fdee10a7f","Type":"ContainerStarted","Data":"fde7a463c69afcf651b65ddfc1fa461628fc094779285932b4aa8178658ca2cf"} Oct 31 03:12:11.119589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.119548 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" event={"ID":"30e269f2-8bf4-41c2-bcb3-426fdee10a7f","Type":"ContainerStarted","Data":"839fbf14fbcac8aaf6c084aaa6542a8b9cf46708fcb6f5d2b5db5c8747f432a4"} Oct 31 03:12:11.233405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.233336 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" podStartSLOduration=3.233313695 podStartE2EDuration="3.233313695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:11.159699384 +0000 UTC m=+1009.014716524" watchObservedRunningTime="2026-10-31 03:12:11.233313695 +0000 UTC m=+1009.088330841" Oct 31 03:12:11.294436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.294385 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229"] Oct 31 03:12:11.595457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.595262 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.800868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.800761 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.801612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.801584 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229"] Oct 31 03:12:11.801956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.801938 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:11.816395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.816364 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:12:11.832338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.829922 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:11.832338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.830011 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:11.832338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.830118 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfm7\" (UniqueName: \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:11.832338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.831209 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.832822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.832797 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:12:11.864361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.864305 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:11.869347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.869302 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.869627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.869608 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:11.929379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.922708 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:12:11.929379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.923055 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:11.929379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.925180 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:11.960440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.953768 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.055852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.055606 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.056240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.056204 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfm7\" (UniqueName: \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.077434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:11.960137 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:12:12.106436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.088728 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.169467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.010970 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.299440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.294984 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfm7\" (UniqueName: \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\") pod \"maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.612210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.612157 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.613636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.613599 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.693440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.678984 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:12:12.693440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.679036 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:12:12.709196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.695502 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 31 03:12:12.737823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.737787 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:12:12.751434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.742293 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.751434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.743828 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:12.825050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.793022 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:12:12.825050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.794036 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:12:12.825050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.794105 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:12:12.825050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.794236 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-97phk\"" Oct 31 03:12:12.852438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.827269 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.853361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.853320 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.853708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.853686 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.854146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.854123 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcswx\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.854286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.854267 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.854494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.854478 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.876474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.854686 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:12.876976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.829626 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.878498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.829662 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:12:12.880039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.879250 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:12.880039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.840770 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:12:12.880614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.880591 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:12:12.880800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.880785 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:12:12.880971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.880957 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:12:12.903789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.874905 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:12.905288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.905259 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:12:12.937441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.927218 4556 generic.go:356] "Generic (PLEG): container finished" podID="30e269f2-8bf4-41c2-bcb3-426fdee10a7f" containerID="fde7a463c69afcf651b65ddfc1fa461628fc094779285932b4aa8178658ca2cf" exitCode=0 Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.030891 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.030946 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.082129 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" event={"ID":"30e269f2-8bf4-41c2-bcb3-426fdee10a7f","Type":"ContainerDied","Data":"fde7a463c69afcf651b65ddfc1fa461628fc094779285932b4aa8178658ca2cf"} Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.082226 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.082268 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.840736 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.994155 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:13.103393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.092674 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:12:13.220130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.105559 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.874938 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:12.874957 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.153113 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.226633 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.226706 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.226849 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.227138 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcswx\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.227175 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.247468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.227259 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.253569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.253532 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.253787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.198440 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:12:13.254061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.254039 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:12:13.254216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.254202 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:12:13.667043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.667003 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:12:13.761438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.674452 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.761438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.735750 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:12:13.761438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.712931 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcswx\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.800353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.789552 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.847553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.847498 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:13.954500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.905509 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.954500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.944035 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:13.997408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:13.997360 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.037340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.023531 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.056926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.056467 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.058305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.057997 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.063520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.060782 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:14.067599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.067562 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:14.105529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.085509 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.178205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.163885 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:14.284498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.267913 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:14.284498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.270603 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:14.284498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.270917 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.371278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.332525 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:12:14.478647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.478610 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:14.735495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.734954 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-442jh\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:14.746585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.745803 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:14.831651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.776372 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:12:15.056837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.942516 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.056837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.969740 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:15.056837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:14.970510 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.056837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.000658 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:12:15.056837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.039528 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.057686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.057638 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.059089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.059056 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.065502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.065471 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.097654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.090204 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:12:15.097814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.097757 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.097909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.097882 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:12:15.377478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.357919 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:12:15.377478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.358009 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:15.377478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.358054 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:15.565127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.516660 4556 patch_prober.go:29] interesting pod/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:12:15.565127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.516729 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.764996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.764957 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:15.853466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.841155 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:12:16.060603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.060523 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:12:16.060835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:15.897709 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.089625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.089579 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:16.093461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.060315 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.113437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.110242 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.158999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.046544 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.292536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.292493 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:12:16.294290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.046590 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.512830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.510934 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:12:16.552445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.542739 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:16.552445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.543041 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:16.803169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.800923 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 31 03:12:16.817212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.817112 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856057 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856159 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856219 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856266 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856439 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:12:16.867538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.856494 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:12:16.881887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.876652 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:12:16.881887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.878111 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:12:16.881887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.878305 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:12:16.881887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.879951 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:12:16.883588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.883563 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:16.955502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:16.955277 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:12:16.973733 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:12:16.973695 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:12:17.051482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.050519 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:17.155635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.155581 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229"] Oct 31 03:12:17.181156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.180235 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\") pod \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " Oct 31 03:12:17.181592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.181568 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\" (UniqueName: \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\") pod \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " Oct 31 03:12:17.182243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.182219 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\") pod \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\" (UID: \"30e269f2-8bf4-41c2-bcb3-426fdee10a7f\") " Oct 31 03:12:17.182642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.182606 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca" pod "30e269f2-8bf4-41c2-bcb3-426fdee10a7f" (UID: "30e269f2-8bf4-41c2-bcb3-426fdee10a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:17.183825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.183249 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:17.186691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.186657 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca" pod "30e269f2-8bf4-41c2-bcb3-426fdee10a7f" (UID: "30e269f2-8bf4-41c2-bcb3-426fdee10a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:17.187525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.187483 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88" pod "30e269f2-8bf4-41c2-bcb3-426fdee10a7f" (UID: "30e269f2-8bf4-41c2-bcb3-426fdee10a7f"). InnerVolumeSpecName "kube-api-access-4rn88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:17.230029 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:12:17.229941 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d0b005_0520_436e_960d_261865375324.slice/crio-cf36417918a010be0d4f4c02c8b80c7ad7332c9275ba129add1af15406702edd WatchSource:0}: Error finding container cf36417918a010be0d4f4c02c8b80c7ad7332c9275ba129add1af15406702edd: Status 404 returned error can't find the container with id cf36417918a010be0d4f4c02c8b80c7ad7332c9275ba129add1af15406702edd Oct 31 03:12:17.281709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.281672 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:12:17.288525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.288464 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:17.288525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.288500 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rn88\" (UniqueName: \"kubernetes.io/projected/30e269f2-8bf4-41c2-bcb3-426fdee10a7f-kube-api-access-4rn88\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:17.315937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.315828 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" Oct 31 03:12:17.315937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.315827 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml" event={"ID":"30e269f2-8bf4-41c2-bcb3-426fdee10a7f","Type":"ContainerDied","Data":"839fbf14fbcac8aaf6c084aaa6542a8b9cf46708fcb6f5d2b5db5c8747f432a4"} Oct 31 03:12:17.316166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.315953 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839fbf14fbcac8aaf6c084aaa6542a8b9cf46708fcb6f5d2b5db5c8747f432a4" Oct 31 03:12:17.318554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.317934 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" event={"ID":"d6d0b005-0520-436e-960d-261865375324","Type":"ContainerStarted","Data":"cf36417918a010be0d4f4c02c8b80c7ad7332c9275ba129add1af15406702edd"} Oct 31 03:12:17.319722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.319694 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" event={"ID":"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a","Type":"ContainerStarted","Data":"fad72c3a3723cf24c60e8c52c56fb8dd61acf53d4f3280c4d4dac00781b89531"} Oct 31 03:12:17.809650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:17.809615 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:18.031537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:18.031364 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:18.332101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:18.332004 4556 generic.go:356] "Generic (PLEG): container finished" podID="bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" containerID="6396a49d4f135a6747b0fa09e3287c6e493e56fa0aba44922fc8fc4271852cfa" exitCode=0 Oct 31 03:12:18.332257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:18.332096 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" event={"ID":"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a","Type":"ContainerDied","Data":"6396a49d4f135a6747b0fa09e3287c6e493e56fa0aba44922fc8fc4271852cfa"} Oct 31 03:12:18.694222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:18.694190 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:19.594551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.594313 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:19.670507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.654035 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:19.767517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.767479 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\") pod \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " Oct 31 03:12:19.767718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.767651 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\" (UniqueName: \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\") pod \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " Oct 31 03:12:19.767805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.767781 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\") pod \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\" (UID: \"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a\") " Oct 31 03:12:19.769315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.769281 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca" pod "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" (UID: "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:19.771399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.771367 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca" pod "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" (UID: "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:19.772555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.772529 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7" pod "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" (UID: "bb789c21-aae8-45bb-a65a-13e7ff3f1f6a"). InnerVolumeSpecName "kube-api-access-jwfm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:19.870098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.870060 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfm7\" (UniqueName: \"kubernetes.io/projected/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-kube-api-access-jwfm7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:19.870098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.870098 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:19.870322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:19.870115 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:20.375386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:20.375205 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" Oct 31 03:12:20.375386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:20.375263 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229" event={"ID":"bb789c21-aae8-45bb-a65a-13e7ff3f1f6a","Type":"ContainerDied","Data":"fad72c3a3723cf24c60e8c52c56fb8dd61acf53d4f3280c4d4dac00781b89531"} Oct 31 03:12:20.375386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:20.375321 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad72c3a3723cf24c60e8c52c56fb8dd61acf53d4f3280c4d4dac00781b89531" Oct 31 03:12:20.583374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:20.583265 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:20.836807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:20.836715 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:21.512445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:21.512027 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:21.670436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:21.670207 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:21.819297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:21.819215 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:12:21.850029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:21.849980 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:12:21.973088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:21.973057 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:22.046458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.046389 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:22.046589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.046484 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:22.113013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.112978 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:22.533995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.533960 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" event={"ID":"d6d0b005-0520-436e-960d-261865375324","Type":"ContainerStarted","Data":"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa"} Oct 31 03:12:22.534169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.534115 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:22.557844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.557794 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" podStartSLOduration=6.745923284 podStartE2EDuration="11.557780273s" totalImagesPullingTime="4.811856989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:22.554733913 +0000 UTC m=+1020.409751056" watchObservedRunningTime="2026-10-31 03:12:22.557780273 +0000 UTC m=+1020.412797413" Oct 31 03:12:22.689604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.689548 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:22.839502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:22.839183 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:23.093147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:23.093061 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:23.545395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:23.545363 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:12:23.716639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:23.716592 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:24.533003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:24.532948 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:24.908283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:24.908248 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:25.085458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.085402 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:25.130982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.130932 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5"] Oct 31 03:12:25.132607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.132578 4556 cpu_manager.go:414] "containerMap: removing container" podUID="bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" containerName="revoke-keys" Oct 31 03:12:25.132802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.132788 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" containerName="revoke-keys" Oct 31 03:12:25.132957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.132946 4556 cpu_manager.go:414] "containerMap: removing container" podUID="30e269f2-8bf4-41c2-bcb3-426fdee10a7f" containerName="revoke-keys" Oct 31 03:12:25.133047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.133037 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e269f2-8bf4-41c2-bcb3-426fdee10a7f" containerName="revoke-keys" Oct 31 03:12:25.139307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.139280 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" containerName="revoke-keys" Oct 31 03:12:25.139538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.139523 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e269f2-8bf4-41c2-bcb3-426fdee10a7f" containerName="revoke-keys" Oct 31 03:12:25.231066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.230973 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.261355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.260756 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:12:25.261355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.260990 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:12:25.269872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.269845 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5"] Oct 31 03:12:25.408409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.408357 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.408602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.408545 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.408602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.408597 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw8x\" (UniqueName: \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.510024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.509930 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw8x\" (UniqueName: \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.510184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.510160 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.510239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.510227 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.510918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.510889 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.512719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.512687 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.521927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.521904 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw8x\" (UniqueName: \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\") pod \"maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.610016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.609972 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:25.826547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.826340 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:25.914136 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:12:25.914106 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db3dd7f_d5a4_4488_be1c_f939283d6a6a.slice/crio-c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249 WatchSource:0}: Error finding container c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249: Status 404 returned error can't find the container with id c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249 Oct 31 03:12:25.916817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:25.916790 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5"] Oct 31 03:12:26.040067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.040037 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:26.616486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.616382 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" event={"ID":"7db3dd7f-d5a4-4488-be1c-f939283d6a6a","Type":"ContainerStarted","Data":"7e0de6790d23b116e65af942abd0ef5f5a727385bfbfd4c6792e6ab4e1dc2a82"} Oct 31 03:12:26.616918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.616488 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" event={"ID":"7db3dd7f-d5a4-4488-be1c-f939283d6a6a","Type":"ContainerStarted","Data":"c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249"} Oct 31 03:12:26.676635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.676573 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" podStartSLOduration=1.676553156 podStartE2EDuration="1.676553156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:26.674548414 +0000 UTC m=+1024.529565556" watchObservedRunningTime="2026-10-31 03:12:26.676553156 +0000 UTC m=+1024.531570299" Oct 31 03:12:26.819836 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.819793 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" podUID="b241f5f0-cfdd-4106-9589-070412908259" containerName="payload-processing" containerID="cri-o://3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7" gracePeriod=25 Oct 31 03:12:26.851022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.850985 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" podUID="47c9ab2f-971a-4f63-b746-a068e5a3e702" containerName="payload-pre-processing" containerID="cri-o://f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8" gracePeriod=25 Oct 31 03:12:26.915736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:26.915695 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:27.177868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.177789 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:12:27.229018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.228977 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:12:27.270399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.270362 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\") pod \"b241f5f0-cfdd-4106-9589-070412908259\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " Oct 31 03:12:27.270654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.270633 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\") pod \"47c9ab2f-971a-4f63-b746-a068e5a3e702\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " Oct 31 03:12:27.270735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.270719 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\" (UniqueName: \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\") pod \"47c9ab2f-971a-4f63-b746-a068e5a3e702\" (UID: \"47c9ab2f-971a-4f63-b746-a068e5a3e702\") " Oct 31 03:12:27.270824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.270809 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\" (UniqueName: \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\") pod \"b241f5f0-cfdd-4106-9589-070412908259\" (UID: \"b241f5f0-cfdd-4106-9589-070412908259\") " Oct 31 03:12:27.275112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.275058 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h" pod "b241f5f0-cfdd-4106-9589-070412908259" (UID: "b241f5f0-cfdd-4106-9589-070412908259"). InnerVolumeSpecName "kube-api-access-7jb5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:27.275502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.275477 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume" pod "b241f5f0-cfdd-4106-9589-070412908259" (UID: "b241f5f0-cfdd-4106-9589-070412908259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:27.275890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.275864 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume" pod "47c9ab2f-971a-4f63-b746-a068e5a3e702" (UID: "47c9ab2f-971a-4f63-b746-a068e5a3e702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:27.278750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.278722 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz" pod "47c9ab2f-971a-4f63-b746-a068e5a3e702" (UID: "47c9ab2f-971a-4f63-b746-a068e5a3e702"). InnerVolumeSpecName "kube-api-access-8fbbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:27.374281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.374239 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241f5f0-cfdd-4106-9589-070412908259-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:27.374281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.374273 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c9ab2f-971a-4f63-b746-a068e5a3e702-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:27.374281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.374286 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbbz\" (UniqueName: \"kubernetes.io/projected/47c9ab2f-971a-4f63-b746-a068e5a3e702-kube-api-access-8fbbz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:27.374567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.374299 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jb5h\" (UniqueName: \"kubernetes.io/projected/b241f5f0-cfdd-4106-9589-070412908259-kube-api-access-7jb5h\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:27.632062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.632021 4556 generic.go:356] "Generic (PLEG): container finished" podID="47c9ab2f-971a-4f63-b746-a068e5a3e702" containerID="f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8" exitCode=0 Oct 31 03:12:27.632671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.632069 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" event={"ID":"47c9ab2f-971a-4f63-b746-a068e5a3e702","Type":"ContainerDied","Data":"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8"} Oct 31 03:12:27.632834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.632756 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" event={"ID":"47c9ab2f-971a-4f63-b746-a068e5a3e702","Type":"ContainerDied","Data":"3be3a4232d74e51c662b20f6f8a86b96c77bceddc09b0417ff70b328c4ca10d1"} Oct 31 03:12:27.632834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.632127 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4" Oct 31 03:12:27.632834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.632797 4556 scope.go:122] "RemoveContainer" containerID="f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8" Oct 31 03:12:27.642493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.642467 4556 generic.go:356] "Generic (PLEG): container finished" podID="7db3dd7f-d5a4-4488-be1c-f939283d6a6a" containerID="7e0de6790d23b116e65af942abd0ef5f5a727385bfbfd4c6792e6ab4e1dc2a82" exitCode=0 Oct 31 03:12:27.642771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.642563 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" event={"ID":"7db3dd7f-d5a4-4488-be1c-f939283d6a6a","Type":"ContainerDied","Data":"7e0de6790d23b116e65af942abd0ef5f5a727385bfbfd4c6792e6ab4e1dc2a82"} Oct 31 03:12:27.644987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.644955 4556 generic.go:356] "Generic (PLEG): container finished" podID="b241f5f0-cfdd-4106-9589-070412908259" containerID="3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7" exitCode=0 Oct 31 03:12:27.645242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.645200 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" event={"ID":"b241f5f0-cfdd-4106-9589-070412908259","Type":"ContainerDied","Data":"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7"} Oct 31 03:12:27.645345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.645255 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" event={"ID":"b241f5f0-cfdd-4106-9589-070412908259","Type":"ContainerDied","Data":"d286c91e6aa6b75586635e003d1483733830a575fccca1d5a83827c76816bc3b"} Oct 31 03:12:27.645481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.645346 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br" Oct 31 03:12:27.657791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.657774 4556 scope.go:122] "RemoveContainer" containerID="f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8" Oct 31 03:12:27.658100 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:12:27.658075 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8\": container with ID starting with f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8 not found: ID does not exist" containerID="f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8" Oct 31 03:12:27.658186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.658108 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8"} err="failed to get container status \"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8\": rpc error: code = NotFound desc = could not find container \"f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8\": container with ID starting with f2692c1ce4e91f91e3b3dcc78a2997607eeec7272ae2abe8d316815c25f913a8 not found: ID does not exist" Oct 31 03:12:27.658186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.658124 4556 scope.go:122] "RemoveContainer" containerID="3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7" Oct 31 03:12:27.669358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.669333 4556 scope.go:122] "RemoveContainer" containerID="3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7" Oct 31 03:12:27.669891 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:12:27.669868 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7\": container with ID starting with 3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7 not found: ID does not exist" containerID="3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7" Oct 31 03:12:27.670056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.669902 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7"} err="failed to get container status \"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7\": rpc error: code = NotFound desc = could not find container \"3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7\": container with ID starting with 3b5f2f2b8c6c83df668648e8c01b5d7a75f491f05d556bd3c63d0c556d5765e7 not found: ID does not exist" Oct 31 03:12:27.677100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.677061 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:12:27.683251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.683227 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8a8b17ff-9fdcbb495-wjfv4"] Oct 31 03:12:27.715422 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.715376 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:12:27.721225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.721202 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8a8b17ff-57c488666f-mm5br"] Oct 31 03:12:27.803950 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:27.802290 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:28.259844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:28.259809 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c9ab2f-971a-4f63-b746-a068e5a3e702" path="/var/lib/kubelet/pods/47c9ab2f-971a-4f63-b746-a068e5a3e702/volumes" Oct 31 03:12:28.260315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:28.260298 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b241f5f0-cfdd-4106-9589-070412908259" path="/var/lib/kubelet/pods/b241f5f0-cfdd-4106-9589-070412908259/volumes" Oct 31 03:12:28.644222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:28.643080 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:28.895732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:28.895666 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:29.004557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.004518 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\" (UniqueName: \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\") pod \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " Oct 31 03:12:29.004736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.004724 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\") pod \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " Oct 31 03:12:29.004781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.004769 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\") pod \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\" (UID: \"7db3dd7f-d5a4-4488-be1c-f939283d6a6a\") " Oct 31 03:12:29.005118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.005090 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca" pod "7db3dd7f-d5a4-4488-be1c-f939283d6a6a" (UID: "7db3dd7f-d5a4-4488-be1c-f939283d6a6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:29.005345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.005329 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:29.006746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.006715 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x" pod "7db3dd7f-d5a4-4488-be1c-f939283d6a6a" (UID: "7db3dd7f-d5a4-4488-be1c-f939283d6a6a"). InnerVolumeSpecName "kube-api-access-xpw8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:29.006854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.006799 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca" pod "7db3dd7f-d5a4-4488-be1c-f939283d6a6a" (UID: "7db3dd7f-d5a4-4488-be1c-f939283d6a6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:29.107462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.107429 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:29.107462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.107461 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpw8x\" (UniqueName: \"kubernetes.io/projected/7db3dd7f-d5a4-4488-be1c-f939283d6a6a-kube-api-access-xpw8x\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:29.500600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.500557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:29.685641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.685613 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" Oct 31 03:12:29.686015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.685634 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5" event={"ID":"7db3dd7f-d5a4-4488-be1c-f939283d6a6a","Type":"ContainerDied","Data":"c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249"} Oct 31 03:12:29.686015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:29.685690 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1234561504ecbdc3d8ca52b091d5b5d4f8e725495b609473d9533b6c3259249" Oct 31 03:12:30.222296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:30.222255 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:30.931315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:30.931275 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:31.438437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.432996 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:12:31.438437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.433239 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" podUID="01c28b01-3711-449c-a93a-69f2254aa1ed" containerName="maas-api" containerID="cri-o://03d67bdd565c2aefdd392060f3f733f2a6a10b3b578a86449211dd69aa442621" gracePeriod=30 Oct 31 03:12:31.518023 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.517994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:31.700768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.690853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:31.891842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.891489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:31.895742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.895693 4556 generic.go:356] "Generic (PLEG): container finished" podID="01c28b01-3711-449c-a93a-69f2254aa1ed" containerID="03d67bdd565c2aefdd392060f3f733f2a6a10b3b578a86449211dd69aa442621" exitCode=0 Oct 31 03:12:31.895898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.895755 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" event={"ID":"01c28b01-3711-449c-a93a-69f2254aa1ed","Type":"ContainerDied","Data":"03d67bdd565c2aefdd392060f3f733f2a6a10b3b578a86449211dd69aa442621"} Oct 31 03:12:31.940938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.940912 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:12:31.981093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.980995 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229"] Oct 31 03:12:31.992343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:31.992304 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8a8b17ff-5v229"] Oct 31 03:12:32.069144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.069107 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:32.096037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.095999 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\" (UniqueName: \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\") pod \"01c28b01-3711-449c-a93a-69f2254aa1ed\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " Oct 31 03:12:32.096254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.096046 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") pod \"01c28b01-3711-449c-a93a-69f2254aa1ed\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " Oct 31 03:12:32.096254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.096243 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\") pod \"01c28b01-3711-449c-a93a-69f2254aa1ed\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " Oct 31 03:12:32.096594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.096572 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\") pod \"01c28b01-3711-449c-a93a-69f2254aa1ed\" (UID: \"01c28b01-3711-449c-a93a-69f2254aa1ed\") " Oct 31 03:12:32.106786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.106727 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca" pod "01c28b01-3711-449c-a93a-69f2254aa1ed" (UID: "01c28b01-3711-449c-a93a-69f2254aa1ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:32.110648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.110602 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs" pod "01c28b01-3711-449c-a93a-69f2254aa1ed" (UID: "01c28b01-3711-449c-a93a-69f2254aa1ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:32.113232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.113192 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p" pod "01c28b01-3711-449c-a93a-69f2254aa1ed" (UID: "01c28b01-3711-449c-a93a-69f2254aa1ed"). InnerVolumeSpecName "kube-api-access-dvr4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:32.113700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.113673 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls" pod "01c28b01-3711-449c-a93a-69f2254aa1ed" (UID: "01c28b01-3711-449c-a93a-69f2254aa1ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:12:32.207987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.207949 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c28b01-3711-449c-a93a-69f2254aa1ed-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:32.207987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.207984 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:32.208242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.207996 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvr4p\" (UniqueName: \"kubernetes.io/projected/01c28b01-3711-449c-a93a-69f2254aa1ed-kube-api-access-dvr4p\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:32.208242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.208006 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c28b01-3711-449c-a93a-69f2254aa1ed-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:32.262492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.261795 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:32.268436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.268261 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb789c21-aae8-45bb-a65a-13e7ff3f1f6a" path="/var/lib/kubelet/pods/bb789c21-aae8-45bb-a65a-13e7ff3f1f6a/volumes" Oct 31 03:12:32.405432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.404650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:32.910021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.909992 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" Oct 31 03:12:32.910203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.910019 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49" event={"ID":"01c28b01-3711-449c-a93a-69f2254aa1ed","Type":"ContainerDied","Data":"dcdd904b5c60b97c6f5d799b92af9f145929c74b945f5927f1ea96d5d33805ce"} Oct 31 03:12:32.910203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.910081 4556 scope.go:122] "RemoveContainer" containerID="03d67bdd565c2aefdd392060f3f733f2a6a10b3b578a86449211dd69aa442621" Oct 31 03:12:32.951574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.950294 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:12:32.958091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.958064 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8a8b17ff-68899479b8-6hd49"] Oct 31 03:12:32.973595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:32.973544 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:33.119165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.119130 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:33.300667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.300588 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:33.721875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.721841 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722757 4556 cpu_manager.go:414] "containerMap: removing container" podUID="47c9ab2f-971a-4f63-b746-a068e5a3e702" containerName="payload-pre-processing" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722782 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c9ab2f-971a-4f63-b746-a068e5a3e702" containerName="payload-pre-processing" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722798 4556 cpu_manager.go:414] "containerMap: removing container" podUID="01c28b01-3711-449c-a93a-69f2254aa1ed" containerName="maas-api" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722805 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c28b01-3711-449c-a93a-69f2254aa1ed" containerName="maas-api" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722843 4556 cpu_manager.go:414] "containerMap: removing container" podUID="7db3dd7f-d5a4-4488-be1c-f939283d6a6a" containerName="revoke-keys" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722850 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db3dd7f-d5a4-4488-be1c-f939283d6a6a" containerName="revoke-keys" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722864 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b241f5f0-cfdd-4106-9589-070412908259" containerName="payload-processing" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.722871 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b241f5f0-cfdd-4106-9589-070412908259" containerName="payload-processing" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.723064 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c9ab2f-971a-4f63-b746-a068e5a3e702" containerName="payload-pre-processing" Oct 31 03:12:33.723072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.723079 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c28b01-3711-449c-a93a-69f2254aa1ed" containerName="maas-api" Oct 31 03:12:33.723722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.723094 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db3dd7f-d5a4-4488-be1c-f939283d6a6a" containerName="revoke-keys" Oct 31 03:12:33.723722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.723104 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b241f5f0-cfdd-4106-9589-070412908259" containerName="payload-processing" Oct 31 03:12:33.734822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.734797 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.742105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.742065 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:12:33.745197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.745168 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-537aca72-gw-openshift-default-dockercfg-j5mgp\"" Oct 31 03:12:33.800555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.800527 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:33.848611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848580 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.848773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848656 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.848773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848696 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.848847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848799 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.848847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848838 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.848971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848951 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.849005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.848989 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.849136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.849120 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26h8w\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.849184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.849168 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.941146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.941106 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.951727 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.952184 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953126 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.952245 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953308 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953354 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953462 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.953519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953498 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953617 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26h8w\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953676 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.953918 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.954328 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.954462 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.955440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.955292 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.981493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.973573 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26h8w\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.981493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.978068 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.981493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.978767 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:33.988444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:33.987593 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\") pod \"e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:34.101750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.098806 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:34.102299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.102110 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:34.274438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.273876 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c28b01-3711-449c-a93a-69f2254aa1ed" path="/var/lib/kubelet/pods/01c28b01-3711-449c-a93a-69f2254aa1ed/volumes" Oct 31 03:12:34.297621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.297577 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:34.451930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.451854 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:34.487050 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:12:34.487003 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8933b_0eb3_4f70_b3d7_e473ae6f4494.slice/crio-24f37a3cbb927d6061a734d4e60e1f0e986a2b67f5d01f6986fb1b9bce8c8b78 WatchSource:0}: Error finding container 24f37a3cbb927d6061a734d4e60e1f0e986a2b67f5d01f6986fb1b9bce8c8b78: Status 404 returned error can't find the container with id 24f37a3cbb927d6061a734d4e60e1f0e986a2b67f5d01f6986fb1b9bce8c8b78 Oct 31 03:12:34.491684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.491507 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:34.491684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.491619 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:34.492189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.491712 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:12:34.492345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.491955 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:34.931368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.931334 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" event={"ID":"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494","Type":"ContainerStarted","Data":"dba447d7c6359a341a7f90dbf99db1bda1099d52bd567e0a6b2e29adadf674a8"} Oct 31 03:12:34.931585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.931391 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" event={"ID":"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494","Type":"ContainerStarted","Data":"24f37a3cbb927d6061a734d4e60e1f0e986a2b67f5d01f6986fb1b9bce8c8b78"} Oct 31 03:12:34.957521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:34.957461 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" podStartSLOduration=1.9574466259999999 podStartE2EDuration="1.957446626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:34.952354143 +0000 UTC m=+1032.807371284" watchObservedRunningTime="2026-10-31 03:12:34.957446626 +0000 UTC m=+1032.812463767" Oct 31 03:12:35.101370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.099852 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:35.107811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.107773 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:35.272518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.272408 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:35.409486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.409406 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:35.940586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.940561 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:35.941726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:35.941698 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:12:36.075321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:36.075290 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:36.466609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:36.462401 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:36.756195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:36.756116 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:36.896226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:36.896186 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:37.166706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.166663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:37.201709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.201671 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:12:37.208220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.208195 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.217392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.213366 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bb8bca19-openshift-default-dockercfg-vprj2\"" Oct 31 03:12:37.222583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.222556 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:12:37.236122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236094 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236219 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236267 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236315 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g97j\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236333 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236427 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236449 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236490 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.236546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.236521 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.327731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.325439 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.347631 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.347744 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.347823 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g97j\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.347850 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.347989 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.348018 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.348086 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.348137 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.353442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.348255 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.361439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.358070 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.375438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.374342 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.375438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.374654 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.375438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.374886 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.375438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.375409 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.511492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.504181 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g97j\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.521185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.521140 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.525475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.522804 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.525475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.523026 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\") pod \"e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.528685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.528659 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:37.796692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.796595 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:37.902734 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:12:37.902697 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde092f44_a5ab_467e_a2de_6673ad6c5c4c.slice/crio-4484572f89ca37f4f7dba0834f44b41a08e10486f1cda26efa7bd2756cb3283f WatchSource:0}: Error finding container 4484572f89ca37f4f7dba0834f44b41a08e10486f1cda26efa7bd2756cb3283f: Status 404 returned error can't find the container with id 4484572f89ca37f4f7dba0834f44b41a08e10486f1cda26efa7bd2756cb3283f Oct 31 03:12:37.906407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.906314 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:37.906532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.906479 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:37.906589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.906551 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:12:37.908245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.908219 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:12:37.963835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.963802 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" event={"ID":"de092f44-a5ab-467e-a2de-6673ad6c5c4c","Type":"ContainerStarted","Data":"4484572f89ca37f4f7dba0834f44b41a08e10486f1cda26efa7bd2756cb3283f"} Oct 31 03:12:37.965903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:37.965875 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:38.738926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:38.738657 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:38.975928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:38.975893 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" event={"ID":"de092f44-a5ab-467e-a2de-6673ad6c5c4c","Type":"ContainerStarted","Data":"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1"} Oct 31 03:12:39.000318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.000220 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" podStartSLOduration=2.000205845 podStartE2EDuration="2.000205845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:12:38.997565459 +0000 UTC m=+1036.852582600" watchObservedRunningTime="2026-10-31 03:12:39.000205845 +0000 UTC m=+1036.855222998" Oct 31 03:12:39.530052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.530021 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:39.535887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.535862 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:39.908955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.908838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:39.987737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.987702 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:39.988746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:39.988723 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:12:40.590485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:40.590450 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:40.726996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:40.726950 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:40.918520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:40.918353 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:41.156436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.153786 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:41.237476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.236982 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5"] Oct 31 03:12:41.250435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.250213 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1adda1ee-7d4r5"] Oct 31 03:12:41.292290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.292242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:41.438483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.434083 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:12:41.438483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.434324 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" podUID="4be131a2-d7bd-421c-9f0f-dc00722268b6" containerName="istio-proxy" containerID="cri-o://1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d" gracePeriod=30 Oct 31 03:12:41.443090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.443060 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:41.669333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:41.669286 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:42.040837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.040756 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:42.232079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.232038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:42.362343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.362224 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db3dd7f-d5a4-4488-be1c-f939283d6a6a" path="/var/lib/kubelet/pods/7db3dd7f-d5a4-4488-be1c-f939283d6a6a/volumes" Oct 31 03:12:42.598478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.598404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:42.764636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.764572 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:42.903203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:42.903170 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:43.731758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:43.731714 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:44.536466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:44.534642 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:45.230292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:45.230260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:45.367780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:45.367721 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:45.939165 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:12:45.939126 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:12:46.161444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.157791 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:46.735295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.735272 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:12:46.847151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847118 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847169 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847203 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847233 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847305 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847355 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847546 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847596 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847652 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:46.847791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847724 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\") pod \"4be131a2-d7bd-421c-9f0f-dc00722268b6\" (UID: \"4be131a2-d7bd-421c-9f0f-dc00722268b6\") " Oct 31 03:12:46.847889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847862 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:12:46.847952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847897 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:46.848278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.847942 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:46.848372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848237 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:46.848727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848705 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.848789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848736 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be131a2-d7bd-421c-9f0f-dc00722268b6-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.848789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848752 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.848789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848766 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.848789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.848782 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.849730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.849698 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:12:46.849870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.849846 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "kube-api-access-brts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:46.849994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.849971 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:12:46.850209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.850189 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token" pod "4be131a2-d7bd-421c-9f0f-dc00722268b6" (UID: "4be131a2-d7bd-421c-9f0f-dc00722268b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:12:46.950912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.950878 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.950912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.950908 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brts7\" (UniqueName: \"kubernetes.io/projected/4be131a2-d7bd-421c-9f0f-dc00722268b6-kube-api-access-brts7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.950912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.950919 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:46.951155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:46.950930 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be131a2-d7bd-421c-9f0f-dc00722268b6-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:12:47.013484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.013038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:47.059270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.059234 4556 generic.go:356] "Generic (PLEG): container finished" podID="4be131a2-d7bd-421c-9f0f-dc00722268b6" containerID="1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d" exitCode=0 Oct 31 03:12:47.059541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.059298 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" Oct 31 03:12:47.059541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.059332 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" event={"ID":"4be131a2-d7bd-421c-9f0f-dc00722268b6","Type":"ContainerDied","Data":"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d"} Oct 31 03:12:47.059541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.059427 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth" event={"ID":"4be131a2-d7bd-421c-9f0f-dc00722268b6","Type":"ContainerDied","Data":"f015fa8e08a0cbeec4598e5b5568a0aaf2ba316dfc057810574e976a98e072f5"} Oct 31 03:12:47.059541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.059475 4556 scope.go:122] "RemoveContainer" containerID="1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d" Oct 31 03:12:47.078211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.078190 4556 scope.go:122] "RemoveContainer" containerID="1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d" Oct 31 03:12:47.078609 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:12:47.078585 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d\": container with ID starting with 1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d not found: ID does not exist" containerID="1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d" Oct 31 03:12:47.078705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.078618 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d"} err="failed to get container status \"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d\": rpc error: code = NotFound desc = could not find container \"1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d\": container with ID starting with 1643865afff2cdbf3c4a7207e5d138d88e4256a30962c6db7d95bf2d0405e51d not found: ID does not exist" Oct 31 03:12:47.109027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.108986 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:12:47.119452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.119405 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1adda1ee-openshift-default-6c48b97f77-wvpth"] Oct 31 03:12:47.845901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:47.845845 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:48.259773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:48.259736 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be131a2-d7bd-421c-9f0f-dc00722268b6" path="/var/lib/kubelet/pods/4be131a2-d7bd-421c-9f0f-dc00722268b6/volumes" Oct 31 03:12:48.637535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:48.637434 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:48.781370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:48.781336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:48.793219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:48.793186 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:12:49.829281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:49.829242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:51.478438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.470936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:51.553600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.553557 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.788608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.788506 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.823438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.819556 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.823438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.819940 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:12:51.859635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.853661 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:51.885540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:51.885488 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.028368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.027540 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.049735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.049624 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:12:52.064579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.064524 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.065244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.065206 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:12:52.093148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.093089 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.172565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.172515 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.377521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.345742 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:12:52.582566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.582508 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.583317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.583279 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:12:52.659655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.659606 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.817440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.816719 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.817654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.817631 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:12:52.844366 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.844315 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:52.845078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.845043 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:12:52.846018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.845995 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:12:52.846223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.846207 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:12:52.876068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.876021 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:12:52.877016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.876986 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:12:53.096498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.073475 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.149447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.096802 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.097381 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.097448 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.097473 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.131876 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.162161 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.162231 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.162290 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.162353 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:52.942395 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:12:53.217438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.187330 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:12:53.268442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.242602 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:12:53.291402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.269128 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:12:53.465525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.465487 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:12:53.466104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.466080 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:12:53.584433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.583966 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:53.802898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.802724 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:12:53.884483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.869825 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.885103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.873738 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.910637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.876456 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.984520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.984461 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:53.987713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:53.987660 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:54.076543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.056505 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:12:54.197745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.182054 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:12:54.197745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.189601 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:12:54.229439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.223263 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:12:54.247513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.247472 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:12:54.254725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.254687 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.295436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.288152 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:12:54.434435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.405077 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:12:54.469538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.469487 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.569973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.569943 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:12:54.570253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.570225 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:12:54.571184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.571166 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:12:54.583227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.582830 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:12:54.591481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.588590 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:12:54.617447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.617206 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:12:54.775999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.774994 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:12:54.781202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.779594 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:12:54.781202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.780713 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:12:54.799352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.798589 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:12:54.897587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.897512 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:12:54.907464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:54.906760 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:12:55.137498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:55.134956 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:56.131441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:56.130767 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:56.840437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:56.831784 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:57.702987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:57.702937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:58.481569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:58.481530 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:12:59.328047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:12:59.326894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:00.102662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:00.102623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:01.192825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:01.192789 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:02.110863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.110821 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:02.191457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.191383 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7"] Oct 31 03:13:02.192323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.192298 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4be131a2-d7bd-421c-9f0f-dc00722268b6" containerName="istio-proxy" Oct 31 03:13:02.192323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.192323 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be131a2-d7bd-421c-9f0f-dc00722268b6" containerName="istio-proxy" Oct 31 03:13:02.192572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.192528 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be131a2-d7bd-421c-9f0f-dc00722268b6" containerName="istio-proxy" Oct 31 03:13:02.197093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.197067 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.200259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.200229 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:13:02.200484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.200464 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:13:02.207309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.207001 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7"] Oct 31 03:13:02.338901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.338862 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.339131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.338970 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkmb\" (UniqueName: \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.339131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.339062 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.440821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.440789 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.441164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.441141 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.441295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.441271 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkmb\" (UniqueName: \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.441788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.441768 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.443269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.443249 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.453318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.453290 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkmb\" (UniqueName: \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\") pod \"maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.518133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.518096 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:02.812154 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:13:02.812118 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26563d32_563e_4985_9474_eba34c5beb8b.slice/crio-0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9 WatchSource:0}: Error finding container 0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9: Status 404 returned error can't find the container with id 0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9 Oct 31 03:13:02.813769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:02.813742 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7"] Oct 31 03:13:03.130277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:03.130155 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:03.431517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:03.431475 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" event={"ID":"26563d32-563e-4985-9474-eba34c5beb8b","Type":"ContainerStarted","Data":"da459e309c93fe7df5a9bd2bb974a2eeeca728185ca682bfc3681cf8bb9e8dfd"} Oct 31 03:13:03.431958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:03.431543 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" event={"ID":"26563d32-563e-4985-9474-eba34c5beb8b","Type":"ContainerStarted","Data":"0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9"} Oct 31 03:13:03.450916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:03.450837 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" podStartSLOduration=1.450824122 podStartE2EDuration="1.450824122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:03.447594783 +0000 UTC m=+1061.302611924" watchObservedRunningTime="2026-10-31 03:13:03.450824122 +0000 UTC m=+1061.305841263" Oct 31 03:13:03.936448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:03.936334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:04.443064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:04.443031 4556 generic.go:356] "Generic (PLEG): container finished" podID="26563d32-563e-4985-9474-eba34c5beb8b" containerID="da459e309c93fe7df5a9bd2bb974a2eeeca728185ca682bfc3681cf8bb9e8dfd" exitCode=0 Oct 31 03:13:04.443505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:04.443092 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" event={"ID":"26563d32-563e-4985-9474-eba34c5beb8b","Type":"ContainerDied","Data":"da459e309c93fe7df5a9bd2bb974a2eeeca728185ca682bfc3681cf8bb9e8dfd"} Oct 31 03:13:04.649438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:04.648949 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:05.473902 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.473770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:05.842581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.842558 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:05.915405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.915369 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\" (UniqueName: \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\") pod \"26563d32-563e-4985-9474-eba34c5beb8b\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " Oct 31 03:13:05.915787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.915764 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\") pod \"26563d32-563e-4985-9474-eba34c5beb8b\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " Oct 31 03:13:05.915939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.915916 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\") pod \"26563d32-563e-4985-9474-eba34c5beb8b\" (UID: \"26563d32-563e-4985-9474-eba34c5beb8b\") " Oct 31 03:13:05.916442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.916384 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca" pod "26563d32-563e-4985-9474-eba34c5beb8b" (UID: "26563d32-563e-4985-9474-eba34c5beb8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:05.917057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.917023 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:05.918362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.918333 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb" pod "26563d32-563e-4985-9474-eba34c5beb8b" (UID: "26563d32-563e-4985-9474-eba34c5beb8b"). InnerVolumeSpecName "kube-api-access-qlkmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:05.922560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:05.922542 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca" pod "26563d32-563e-4985-9474-eba34c5beb8b" (UID: "26563d32-563e-4985-9474-eba34c5beb8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:06.018269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.018185 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26563d32-563e-4985-9474-eba34c5beb8b-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:06.018269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.018243 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlkmb\" (UniqueName: \"kubernetes.io/projected/26563d32-563e-4985-9474-eba34c5beb8b-kube-api-access-qlkmb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:06.190212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.190175 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:06.498369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.498339 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" Oct 31 03:13:06.498787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.498364 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7" event={"ID":"26563d32-563e-4985-9474-eba34c5beb8b","Type":"ContainerDied","Data":"0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9"} Oct 31 03:13:06.498787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.498460 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8072c89ab9bbd5fd47aa7580660944e309b4e91abe46260ea1ffa0a6d193b9" Oct 31 03:13:06.938518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.938437 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:06.974588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.972537 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc"] Oct 31 03:13:06.974588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.973698 4556 cpu_manager.go:414] "containerMap: removing container" podUID="26563d32-563e-4985-9474-eba34c5beb8b" containerName="revoke-keys" Oct 31 03:13:06.974588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.973722 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26563d32-563e-4985-9474-eba34c5beb8b" containerName="revoke-keys" Oct 31 03:13:06.974588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.973888 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="26563d32-563e-4985-9474-eba34c5beb8b" containerName="revoke-keys" Oct 31 03:13:06.979051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.979026 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:06.981855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.981831 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc"] Oct 31 03:13:06.984333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.983579 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:13:06.984630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:06.984610 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:13:07.139756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.139716 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.139920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.139900 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.140017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.140004 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sw8\" (UniqueName: \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.241199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.241113 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.241357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.241269 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.241442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.241384 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sw8\" (UniqueName: \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.241990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.241936 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.243628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.243607 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.252934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.252908 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sw8\" (UniqueName: \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\") pod \"maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.305083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.305051 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:07.589941 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:13:07.589904 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1af6347_8c99_4f07_94ec_e7b305377a28.slice/crio-6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3 WatchSource:0}: Error finding container 6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3: Status 404 returned error can't find the container with id 6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3 Oct 31 03:13:07.595456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.595395 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc"] Oct 31 03:13:07.911047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:07.910998 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:08.184535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:08.184453 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:08.517306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:08.517215 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" event={"ID":"f1af6347-8c99-4f07-94ec-e7b305377a28","Type":"ContainerStarted","Data":"ca73e6fc4cf0ed8df770a6d3d8b1faf3cead38524097c1a8ff582ca973a42746"} Oct 31 03:13:08.517306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:08.517280 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" event={"ID":"f1af6347-8c99-4f07-94ec-e7b305377a28","Type":"ContainerStarted","Data":"6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3"} Oct 31 03:13:08.541798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:08.541743 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" podStartSLOduration=2.541728859 podStartE2EDuration="2.541728859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:08.536334419 +0000 UTC m=+1066.391351560" watchObservedRunningTime="2026-10-31 03:13:08.541728859 +0000 UTC m=+1066.396746000" Oct 31 03:13:08.647345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:08.647253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:09.284565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:09.284529 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:09.449947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:09.449895 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:09.532950 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:09.532909 4556 generic.go:356] "Generic (PLEG): container finished" podID="f1af6347-8c99-4f07-94ec-e7b305377a28" containerID="ca73e6fc4cf0ed8df770a6d3d8b1faf3cead38524097c1a8ff582ca973a42746" exitCode=0 Oct 31 03:13:09.533130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:09.533001 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" event={"ID":"f1af6347-8c99-4f07-94ec-e7b305377a28","Type":"ContainerDied","Data":"ca73e6fc4cf0ed8df770a6d3d8b1faf3cead38524097c1a8ff582ca973a42746"} Oct 31 03:13:10.136509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.136407 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:10.708758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.708729 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:10.809816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.809732 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\") pod \"f1af6347-8c99-4f07-94ec-e7b305377a28\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " Oct 31 03:13:10.809984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.809924 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\" (UniqueName: \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\") pod \"f1af6347-8c99-4f07-94ec-e7b305377a28\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " Oct 31 03:13:10.810050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.810008 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\") pod \"f1af6347-8c99-4f07-94ec-e7b305377a28\" (UID: \"f1af6347-8c99-4f07-94ec-e7b305377a28\") " Oct 31 03:13:10.810162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.810129 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca" pod "f1af6347-8c99-4f07-94ec-e7b305377a28" (UID: "f1af6347-8c99-4f07-94ec-e7b305377a28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:10.810686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.810658 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:10.812113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.812089 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8" pod "f1af6347-8c99-4f07-94ec-e7b305377a28" (UID: "f1af6347-8c99-4f07-94ec-e7b305377a28"). InnerVolumeSpecName "kube-api-access-l8sw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:10.812232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.812106 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca" pod "f1af6347-8c99-4f07-94ec-e7b305377a28" (UID: "f1af6347-8c99-4f07-94ec-e7b305377a28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:10.821512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.817204 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:10.912010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.911977 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1af6347-8c99-4f07-94ec-e7b305377a28-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:10.912010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:10.912007 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sw8\" (UniqueName: \"kubernetes.io/projected/f1af6347-8c99-4f07-94ec-e7b305377a28-kube-api-access-l8sw8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:11.524300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:11.524264 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:11.554262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:11.554234 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" Oct 31 03:13:11.554458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:11.554266 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc" event={"ID":"f1af6347-8c99-4f07-94ec-e7b305377a28","Type":"ContainerDied","Data":"6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3"} Oct 31 03:13:11.554458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:11.554309 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ead9a129ccb2b60626352d81f6b9d8abaae81216dbc3fca121e6a93064383e3" Oct 31 03:13:12.213437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:12.211947 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:12.493380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:12.493301 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:12.641232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:12.633211 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:12.914583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:12.914543 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:13.103570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.103539 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:13.201134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.201017 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7"] Oct 31 03:13:13.260883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.260845 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-537aca72-jqpn7"] Oct 31 03:13:13.306152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.306113 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:13.415254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.415211 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:13:13.416159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.415576 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" podUID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" containerName="istio-proxy" containerID="cri-o://dba447d7c6359a341a7f90dbf99db1bda1099d52bd567e0a6b2e29adadf674a8" gracePeriod=30 Oct 31 03:13:13.457943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.457849 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:13.617439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:13.610209 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:14.045058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.045016 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc"] Oct 31 03:13:14.051911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.051880 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc7e8c13-r8wtc"] Oct 31 03:13:14.057891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.057855 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8"] Oct 31 03:13:14.063483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.063460 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e666809-cdkc8"] Oct 31 03:13:14.259880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.259845 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26563d32-563e-4985-9474-eba34c5beb8b" path="/var/lib/kubelet/pods/26563d32-563e-4985-9474-eba34c5beb8b/volumes" Oct 31 03:13:14.260487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.260469 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43901272-4ba1-4a18-b2a2-1f22725be21c" path="/var/lib/kubelet/pods/43901272-4ba1-4a18-b2a2-1f22725be21c/volumes" Oct 31 03:13:14.260958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.260941 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e465a17-e093-4986-8b56-5005c318a957" path="/var/lib/kubelet/pods/5e465a17-e093-4986-8b56-5005c318a957/volumes" Oct 31 03:13:14.296818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.288347 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:14.422800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.422745 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:14.819244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.819209 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:14.963907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:14.963873 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:15.296771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:15.296737 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:15.936917 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:13:15.936884 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:13:16.143438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:16.139888 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:16.941252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:16.941213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:17.250703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:17.250623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:17.397843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:17.397803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:17.899656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:17.899611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:18.712347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.712274 4556 generic.go:356] "Generic (PLEG): container finished" podID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" containerID="dba447d7c6359a341a7f90dbf99db1bda1099d52bd567e0a6b2e29adadf674a8" exitCode=0 Oct 31 03:13:18.712813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.712555 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" event={"ID":"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494","Type":"ContainerDied","Data":"dba447d7c6359a341a7f90dbf99db1bda1099d52bd567e0a6b2e29adadf674a8"} Oct 31 03:13:18.738754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.738663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:18.788710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.788677 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:13:18.901665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.901626 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.901864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.901716 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.901864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.901807 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.901993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.901947 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.901993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.901962 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:18.902105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902047 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.902165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902149 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.902226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902189 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.902281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902233 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.902335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902277 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:18.902388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902355 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\") pod \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\" (UID: \"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494\") " Oct 31 03:13:18.902510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902452 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:18.902627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.902610 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:18.903098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.903079 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.903221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.903102 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.903221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.903115 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.903221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.903129 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:18.903408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.903382 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:18.904658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.904608 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:13:18.905887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.905859 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:18.906507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.906476 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "kube-api-access-26h8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:18.910225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:18.910203 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token" pod "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" (UID: "9ae8933b-0eb3-4f70-b3d7-e473ae6f4494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:19.005039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.004964 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:19.005039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.004992 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:19.005039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.005002 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26h8w\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-kube-api-access-26h8w\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:19.005039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.005012 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:19.005039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.005021 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:19.015195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.015164 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:19.149044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.149013 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:19.633357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.633307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:19.726521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.726484 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" event={"ID":"9ae8933b-0eb3-4f70-b3d7-e473ae6f4494","Type":"ContainerDied","Data":"24f37a3cbb927d6061a734d4e60e1f0e986a2b67f5d01f6986fb1b9bce8c8b78"} Oct 31 03:13:19.727016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.726546 4556 scope.go:122] "RemoveContainer" containerID="dba447d7c6359a341a7f90dbf99db1bda1099d52bd567e0a6b2e29adadf674a8" Oct 31 03:13:19.727016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.726686 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst" Oct 31 03:13:19.770263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.770230 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:13:19.776852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:19.776825 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-537aca72-gw-openshift-default-7c79bbfd6f-5nlst"] Oct 31 03:13:20.216257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:20.216221 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:20.261802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:20.261767 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" path="/var/lib/kubelet/pods/9ae8933b-0eb3-4f70-b3d7-e473ae6f4494/volumes" Oct 31 03:13:20.355763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:20.355728 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:20.520487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:20.512431 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:21.160103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:21.160062 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:21.838771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:21.837642 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:22.030831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:22.030795 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd"] Oct 31 03:13:22.036689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:22.036668 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c24efc-4n4cd"] Oct 31 03:13:22.263076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:22.263042 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a00190c-28a7-4956-b553-b81f6b5e5c96" path="/var/lib/kubelet/pods/0a00190c-28a7-4956-b553-b81f6b5e5c96/volumes" Oct 31 03:13:22.644860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:22.644745 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:22.960341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:22.960299 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:13:23.481430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:23.481359 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:23.716305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:23.714012 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:23.870826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:23.870731 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc"] Oct 31 03:13:23.878145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:23.878109 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb8bca19-zwjgc"] Oct 31 03:13:23.924436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:23.915223 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:24.032283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.032248 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74"] Oct 31 03:13:24.038892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.038864 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj"] Oct 31 03:13:24.044735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.044706 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3c98028-qfd74"] Oct 31 03:13:24.050378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.050353 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d6858d9-sb9xj"] Oct 31 03:13:24.064453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.064398 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:24.363897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.363859 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385dcfea-78ec-4f66-b6ff-da6e7aaef53f" path="/var/lib/kubelet/pods/385dcfea-78ec-4f66-b6ff-da6e7aaef53f/volumes" Oct 31 03:13:24.365666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.365579 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6add1fc1-c5a8-4308-bc71-1b8e7cccf270" path="/var/lib/kubelet/pods/6add1fc1-c5a8-4308-bc71-1b8e7cccf270/volumes" Oct 31 03:13:24.366709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.366686 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1af6347-8c99-4f07-94ec-e7b305377a28" path="/var/lib/kubelet/pods/f1af6347-8c99-4f07-94ec-e7b305377a28/volumes" Oct 31 03:13:24.593795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.593765 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:13:24.594488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.593991 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerName="istio-proxy" containerID="cri-o://8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1" gracePeriod=30 Oct 31 03:13:24.681504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.681393 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:24.995878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.992061 4556 patch_prober.go:29] interesting pod/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:13:24.995878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:24.992166 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:13:25.103210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:25.103161 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:25.278494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:25.271367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:25.557926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:25.557777 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:26.146263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:26.146195 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:26.494469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:26.482927 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:26.664450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:26.663883 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:27.698438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:27.696795 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:28.939164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:28.939095 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:29.719392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:29.719363 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:29.937094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:29.937068 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:13:30.005376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005346 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005398 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005476 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005586 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005722 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005770 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.005831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005780 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005844 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005892 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.005914 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006069 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006115 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\") pod \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\" (UID: \"de092f44-a5ab-467e-a2de-6673ad6c5c4c\") " Oct 31 03:13:30.006177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006153 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:30.006576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006202 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:30.006961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006936 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.007065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006967 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.007065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.006984 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.007065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.007000 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.007065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.007013 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.007975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.007946 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "kube-api-access-2g97j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:30.008122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.008099 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:30.008250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.008231 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:13:30.008354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.008340 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo" pod "de092f44-a5ab-467e-a2de-6673ad6c5c4c" (UID: "de092f44-a5ab-467e-a2de-6673ad6c5c4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:13:30.084917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.084870 4556 generic.go:356] "Generic (PLEG): container finished" podID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerID="8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1" exitCode=0 Oct 31 03:13:30.085105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.084925 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" Oct 31 03:13:30.085105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.084952 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" event={"ID":"de092f44-a5ab-467e-a2de-6673ad6c5c4c","Type":"ContainerDied","Data":"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1"} Oct 31 03:13:30.085105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.085030 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw" event={"ID":"de092f44-a5ab-467e-a2de-6673ad6c5c4c","Type":"ContainerDied","Data":"4484572f89ca37f4f7dba0834f44b41a08e10486f1cda26efa7bd2756cb3283f"} Oct 31 03:13:30.085105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.085060 4556 scope.go:122] "RemoveContainer" containerID="8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1" Oct 31 03:13:30.108484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.108459 4556 scope.go:122] "RemoveContainer" containerID="8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1" Oct 31 03:13:30.108883 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:13:30.108853 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1\": container with ID starting with 8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1 not found: ID does not exist" containerID="8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1" Oct 31 03:13:30.108980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.108898 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1"} err="failed to get container status \"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1\": rpc error: code = NotFound desc = could not find container \"8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1\": container with ID starting with 8e12f45879f2925602686373f7f1d114ee724110aabe58101b76a7fdc7bddcb1 not found: ID does not exist" Oct 31 03:13:30.109311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.109287 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.109383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.109317 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.109383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.109332 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g97j\" (UniqueName: \"kubernetes.io/projected/de092f44-a5ab-467e-a2de-6673ad6c5c4c-kube-api-access-2g97j\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.109383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.109345 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de092f44-a5ab-467e-a2de-6673ad6c5c4c-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:30.125549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.125517 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:13:30.132344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.132313 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bb8bca19-openshift-default-65b4f8fb57-m44lw"] Oct 31 03:13:30.260770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.260735 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" path="/var/lib/kubelet/pods/de092f44-a5ab-467e-a2de-6673ad6c5c4c/volumes" Oct 31 03:13:30.637726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:30.636346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:31.467003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:31.456796 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:32.257566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:32.257520 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:32.728319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:32.728283 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:32.864757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:32.864720 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:33.032732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.032650 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6"] Oct 31 03:13:33.038280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.038254 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v"] Oct 31 03:13:33.043102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.043078 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44bc2c48-8xwd6"] Oct 31 03:13:33.047775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.047745 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c886043-b6p8v"] Oct 31 03:13:33.136290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.136203 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:33.860118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.860082 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:33.994921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:33.994888 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:34.310196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:34.310153 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4b62f0-d4a3-43c1-a351-bfa28deebc86" path="/var/lib/kubelet/pods/0c4b62f0-d4a3-43c1-a351-bfa28deebc86/volumes" Oct 31 03:13:34.311142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:34.311109 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6836a2-3bca-4deb-a368-2da68208bb1e" path="/var/lib/kubelet/pods/9f6836a2-3bca-4deb-a368-2da68208bb1e/volumes" Oct 31 03:13:34.941021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:34.940954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:35.031105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:35.031066 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx"] Oct 31 03:13:35.037246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:35.037211 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6499afd5-px9wx"] Oct 31 03:13:35.731029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:35.730689 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:36.259133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:36.259095 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf49352-b6c8-49f1-9ef6-a6da94185c2a" path="/var/lib/kubelet/pods/baf49352-b6c8-49f1-9ef6-a6da94185c2a/volumes" Oct 31 03:13:36.415515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:36.415477 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:37.128537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:37.127969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:37.851526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:37.851488 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:38.034887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.034843 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv"] Oct 31 03:13:38.040740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.040712 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4523c56-w84wv"] Oct 31 03:13:38.045761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.045731 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf"] Oct 31 03:13:38.050686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.050661 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c4a02f7-gkbhf"] Oct 31 03:13:38.260297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.260262 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743d216f-e4b8-4e66-9cab-ff6ed5378920" path="/var/lib/kubelet/pods/743d216f-e4b8-4e66-9cab-ff6ed5378920/volumes" Oct 31 03:13:38.260848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.260830 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb" path="/var/lib/kubelet/pods/9c9af01a-9dcf-4c0e-a70d-da0a7a18bdfb/volumes" Oct 31 03:13:38.538472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:38.538298 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:39.142315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.142279 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:13:39.143898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143865 4556 cpu_manager.go:414] "containerMap: removing container" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerName="istio-proxy" Oct 31 03:13:39.143898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143893 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerName="istio-proxy" Oct 31 03:13:39.144103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143907 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" containerName="istio-proxy" Oct 31 03:13:39.144103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143915 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" containerName="istio-proxy" Oct 31 03:13:39.144103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143925 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f1af6347-8c99-4f07-94ec-e7b305377a28" containerName="revoke-keys" Oct 31 03:13:39.144103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.143932 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1af6347-8c99-4f07-94ec-e7b305377a28" containerName="revoke-keys" Oct 31 03:13:39.144103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.144097 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="de092f44-a5ab-467e-a2de-6673ad6c5c4c" containerName="istio-proxy" Oct 31 03:13:39.144577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.144109 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae8933b-0eb3-4f70-b3d7-e473ae6f4494" containerName="istio-proxy" Oct 31 03:13:39.144577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.144119 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1af6347-8c99-4f07-94ec-e7b305377a28" containerName="revoke-keys" Oct 31 03:13:39.167645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.167604 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:13:39.167834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.167700 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.172116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.172081 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-fd2e0b-serving-cert\"" Oct 31 03:13:39.245591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.245554 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:39.298952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.298910 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.299147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.299064 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.299242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.299220 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.299302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.299283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtdw\" (UniqueName: \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.400645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.400610 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.400861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.400776 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.400944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.400920 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.401001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.400967 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtdw\" (UniqueName: \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.401651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.401624 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.403403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.403380 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.403518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.403473 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.412081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.412030 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:39.413084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.413060 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtdw\" (UniqueName: \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\") pod \"maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.501372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.501344 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:39.788025 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:13:39.787994 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode014c245_4973_4be6_9968_2bb75c5fda94.slice/crio-a74d50e1c01654d573cd826d266db5cdc3d94888aa787892c0f56d0b6280ff9c WatchSource:0}: Error finding container a74d50e1c01654d573cd826d266db5cdc3d94888aa787892c0f56d0b6280ff9c: Status 404 returned error can't find the container with id a74d50e1c01654d573cd826d266db5cdc3d94888aa787892c0f56d0b6280ff9c Oct 31 03:13:39.789469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:39.789444 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:13:40.241642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:40.241615 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" event={"ID":"e014c245-4973-4be6-9968-2bb75c5fda94","Type":"ContainerStarted","Data":"a74d50e1c01654d573cd826d266db5cdc3d94888aa787892c0f56d0b6280ff9c"} Oct 31 03:13:40.381820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:40.381782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:41.164309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:41.164274 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:41.253715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:41.253683 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" event={"ID":"e014c245-4973-4be6-9968-2bb75c5fda94","Type":"ContainerStarted","Data":"918518a4ea6e6be774790ff32748bdffa35e717b08f7e9512b820022cdd37da6"} Oct 31 03:13:41.254174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:41.253756 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:41.276208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:41.276161 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" podStartSLOduration=1.826441025 podStartE2EDuration="2.276147214s" totalImagesPullingTime="449.706189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:41.272859923 +0000 UTC m=+1099.127877065" watchObservedRunningTime="2026-10-31 03:13:41.276147214 +0000 UTC m=+1099.131164361" Oct 31 03:13:41.843384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:41.843344 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:42.789605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:42.789567 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:43.530224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:43.530186 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:44.064528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.064496 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w"] Oct 31 03:13:44.070583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.070549 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.082876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.081518 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:13:44.082876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.081704 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:13:44.082876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.082026 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w"] Oct 31 03:13:44.184531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.184488 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmsvx\" (UniqueName: \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.184728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.184553 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.184820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.184793 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.312435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.288873 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmsvx\" (UniqueName: \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.312435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.288934 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.312435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.289270 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.329854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.329764 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.349862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.349826 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.365769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.365729 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:44.378617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.378558 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmsvx\" (UniqueName: \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\") pod \"maas-api-revoke-sub-d06eae1b-tbt8w\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.400435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.399570 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:44.536438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.534489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:44.847343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.847283 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w"] Oct 31 03:13:44.924439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:44.913371 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:45.276605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.276522 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:13:45.283195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.283167 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.302049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.302021 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:13:45.315044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.315014 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3fc2d0-openshift-default-dockercfg-lt5nq\"" Oct 31 03:13:45.357645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.357610 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:45.370230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.370191 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.370468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.370248 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.371733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.371700 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.371933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.371912 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.372022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.371976 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbptb\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.372084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.372016 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.372084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.372052 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.372740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.372718 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.373140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.373119 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.460469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.460429 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" event={"ID":"a91abd35-c4ed-4285-b927-5d31237b52a1","Type":"ContainerStarted","Data":"976cb05e5901d5238a935de2bc2ad7a0c6a07c6eba560555e46dfcf413267890"} Oct 31 03:13:45.474625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474582 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.474791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474682 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.474791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474716 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.474930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474842 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.474930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474889 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474941 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbptb\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.474979 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.475011 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.475051 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.475202 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.475372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.475352 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.477787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.475678 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.477787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.476127 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.477787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.476601 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.498186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.479536 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.498186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.479885 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.530300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.512975 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:45.543148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.537817 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbptb\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.557434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.548003 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\") pod \"e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.605430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:45.604813 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:45.948053 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:13:45.948019 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:13:46.029248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.029220 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df"] Oct 31 03:13:46.031754 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:13:46.031725 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07829292_1d8e_4032_91e6_6fa8d3fffcb3.slice/crio-ec3b48cf6cf48044b453a4ca535e2acfb53e06ceff8d135280c8490353277896 WatchSource:0}: Error finding container ec3b48cf6cf48044b453a4ca535e2acfb53e06ceff8d135280c8490353277896: Status 404 returned error can't find the container with id ec3b48cf6cf48044b453a4ca535e2acfb53e06ceff8d135280c8490353277896 Oct 31 03:13:46.034666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.034631 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:46.034748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.034701 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:46.034748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.034736 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:46.035459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.035440 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:13:46.041700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.041669 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af8cb67-8p9df"] Oct 31 03:13:46.265225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.265192 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a473a4dd-7bb2-46f1-9da5-ac914210c1d6" path="/var/lib/kubelet/pods/a473a4dd-7bb2-46f1-9da5-ac914210c1d6/volumes" Oct 31 03:13:46.316748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.316715 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:46.474465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.474365 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerStarted","Data":"769d6fafdab299030aebc27bb6dc5bb8e8f6562186848fec51ab34cfd13c0222"} Oct 31 03:13:46.474717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.474694 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerStarted","Data":"ec3b48cf6cf48044b453a4ca535e2acfb53e06ceff8d135280c8490353277896"} Oct 31 03:13:46.476583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.476555 4556 generic.go:356] "Generic (PLEG): container finished" podID="a91abd35-c4ed-4285-b927-5d31237b52a1" containerID="2d7799deefe51ad072d445d5c667c40b310f5dca13156fddcdf441d36c962029" exitCode=0 Oct 31 03:13:46.476677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.476617 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" event={"ID":"a91abd35-c4ed-4285-b927-5d31237b52a1","Type":"ContainerDied","Data":"2d7799deefe51ad072d445d5c667c40b310f5dca13156fddcdf441d36c962029"} Oct 31 03:13:46.495808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.495756 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podStartSLOduration=1.4957419060000001 podStartE2EDuration="1.495741906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:46.495310338 +0000 UTC m=+1104.350327480" watchObservedRunningTime="2026-10-31 03:13:46.495741906 +0000 UTC m=+1104.350759047" Oct 31 03:13:46.606673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:46.606588 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:47.125841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.125796 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:47.270034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.270003 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:13:47.613861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.613827 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:47.669403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.669374 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:47.746300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.746199 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\" (UniqueName: \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\") pod \"a91abd35-c4ed-4285-b927-5d31237b52a1\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " Oct 31 03:13:47.746300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.746273 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\") pod \"a91abd35-c4ed-4285-b927-5d31237b52a1\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " Oct 31 03:13:47.746592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.746486 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\") pod \"a91abd35-c4ed-4285-b927-5d31237b52a1\" (UID: \"a91abd35-c4ed-4285-b927-5d31237b52a1\") " Oct 31 03:13:47.747220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.747185 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca" pod "a91abd35-c4ed-4285-b927-5d31237b52a1" (UID: "a91abd35-c4ed-4285-b927-5d31237b52a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:13:47.747714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.747682 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:47.749584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.749559 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx" pod "a91abd35-c4ed-4285-b927-5d31237b52a1" (UID: "a91abd35-c4ed-4285-b927-5d31237b52a1"). InnerVolumeSpecName "kube-api-access-dmsvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:13:47.749584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.749570 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca" pod "a91abd35-c4ed-4285-b927-5d31237b52a1" (UID: "a91abd35-c4ed-4285-b927-5d31237b52a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:13:47.849375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.849332 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmsvx\" (UniqueName: \"kubernetes.io/projected/a91abd35-c4ed-4285-b927-5d31237b52a1-kube-api-access-dmsvx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:47.849375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:47.849368 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a91abd35-c4ed-4285-b927-5d31237b52a1-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:13:48.170061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.170025 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:48.314345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.314306 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:48.497167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.497088 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" Oct 31 03:13:48.497167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.497106 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w" event={"ID":"a91abd35-c4ed-4285-b927-5d31237b52a1","Type":"ContainerDied","Data":"976cb05e5901d5238a935de2bc2ad7a0c6a07c6eba560555e46dfcf413267890"} Oct 31 03:13:48.497386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.497187 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="976cb05e5901d5238a935de2bc2ad7a0c6a07c6eba560555e46dfcf413267890" Oct 31 03:13:48.497581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.497556 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:48.498719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:48.498697 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:13:49.005318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:49.005277 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:49.155437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:49.155036 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:49.291533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:49.289538 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:50.029151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:50.029102 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5"] Oct 31 03:13:50.035146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:50.035118 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e288b98-gn4b5"] Oct 31 03:13:50.208492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:50.208454 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:50.262798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:50.262765 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa56fd66-7bb4-4a66-afb5-9222fccd3294" path="/var/lib/kubelet/pods/aa56fd66-7bb4-4a66-afb5-9222fccd3294/volumes" Oct 31 03:13:50.414128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:50.394609 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:51.029193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:51.029161 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g"] Oct 31 03:13:51.034491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:51.034465 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31f19e94-2z87g"] Oct 31 03:13:51.207470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:51.197046 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:52.022946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:52.022908 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:52.293519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:52.293432 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c451252-2a22-44a0-823f-1e5813536fde" path="/var/lib/kubelet/pods/6c451252-2a22-44a0-823f-1e5813536fde/volumes" Oct 31 03:13:52.856581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:52.856544 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:53.013974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.013865 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:53.434080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.434030 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:53.623292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.623232 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:53.852708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.852626 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:13:53.854052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.854019 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a91abd35-c4ed-4285-b927-5d31237b52a1" containerName="revoke-keys" Oct 31 03:13:53.854052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.854051 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91abd35-c4ed-4285-b927-5d31237b52a1" containerName="revoke-keys" Oct 31 03:13:53.854256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.854240 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91abd35-c4ed-4285-b927-5d31237b52a1" containerName="revoke-keys" Oct 31 03:13:53.876805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.876772 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.881786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.881759 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-677281-openshift-default-dockercfg-2vqqr\"" Oct 31 03:13:53.882004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.881983 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:13:53.906143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.905732 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.906143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.905866 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.906143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.906004 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhbs\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.906438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.906148 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.906438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.906375 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.912563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.912530 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.912687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.912591 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.912687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.912618 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.912816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.912726 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:53.934234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:53.934198 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040145 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040325 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040402 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040570 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhbs\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040696 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040869 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040929 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040969 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.040998 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.041818 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.042072 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.042598 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.042831 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.043435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.043340 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.044681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.044231 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.072618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.072398 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.078628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.078552 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhbs\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.082066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.082000 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\") pod \"e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.136713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.134750 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:54.195802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.195771 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:54.411456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.410599 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:54.498523 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:13:54.498485 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263b74e5_a37c_4823_8232_317ea36efd3e.slice/crio-9a5c5d0ce2544b0b8aea433f26f47e5d0f1009ecab0c013d5bab0ac872e5638e WatchSource:0}: Error finding container 9a5c5d0ce2544b0b8aea433f26f47e5d0f1009ecab0c013d5bab0ac872e5638e: Status 404 returned error can't find the container with id 9a5c5d0ce2544b0b8aea433f26f47e5d0f1009ecab0c013d5bab0ac872e5638e Oct 31 03:13:54.499933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.499909 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:13:54.501266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.501236 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:54.501336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.501325 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:54.501378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.501358 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:13:54.557672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.557599 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:54.662709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.662611 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerStarted","Data":"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46"} Oct 31 03:13:54.662887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.662708 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerStarted","Data":"9a5c5d0ce2544b0b8aea433f26f47e5d0f1009ecab0c013d5bab0ac872e5638e"} Oct 31 03:13:54.687903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:54.687850 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podStartSLOduration=1.687821718 podStartE2EDuration="1.687821718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:54.683604516 +0000 UTC m=+1112.538621659" watchObservedRunningTime="2026-10-31 03:13:54.687821718 +0000 UTC m=+1112.542838859" Oct 31 03:13:55.196124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:55.196094 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:55.202387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:55.202357 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:55.442888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:55.442815 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:55.671678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:55.671642 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:55.672837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:55.672814 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:13:56.031016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:56.030939 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj"] Oct 31 03:13:56.036695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:56.036665 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e1914d-pfwmj"] Oct 31 03:13:56.194069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:56.192570 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:56.263844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:56.263809 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70c7ad8-7b52-4fca-8715-ccb18abe117c" path="/var/lib/kubelet/pods/e70c7ad8-7b52-4fca-8715-ccb18abe117c/volumes" Oct 31 03:13:57.208712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:57.208682 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:57.409539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:57.406801 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:58.417181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.417135 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:58.831814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.831732 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:58.874134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.874092 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:13:58.881088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.881044 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:58.886066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.886039 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-677281-dockercfg-b5qmr\"" Oct 31 03:13:58.891159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.891130 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:13:58.953079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.953049 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-677281\"" Oct 31 03:13:58.957377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.957346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:58.976608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.976558 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:58.977143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:58.977049 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq746\" (UniqueName: \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.080752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.080707 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.082019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.081961 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq746\" (UniqueName: \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.082989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.081850 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.113473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.109866 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq746\" (UniqueName: \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\") pod \"payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.214794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.214751 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.526641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.523774 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:13:59.526641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.524012 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:13:59.711344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.711313 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" event={"ID":"c0e6036e-25e1-4103-ba8b-4d3f4048db2a","Type":"ContainerStarted","Data":"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158"} Oct 31 03:13:59.711532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.711371 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" event={"ID":"c0e6036e-25e1-4103-ba8b-4d3f4048db2a","Type":"ContainerStarted","Data":"63bd08cd31dcb4acd9b4ee743c3a68e498d0bdb5555a5f9138861a8a1eb344e8"} Oct 31 03:13:59.711532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.711399 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:13:59.730873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:13:59.730826 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" podStartSLOduration=1.730813476 podStartE2EDuration="1.730813476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:59.726928931 +0000 UTC m=+1117.581946074" watchObservedRunningTime="2026-10-31 03:13:59.730813476 +0000 UTC m=+1117.585830617" Oct 31 03:14:00.029122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:00.029085 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg"] Oct 31 03:14:00.034619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:00.034590 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-174f3f14-749bg"] Oct 31 03:14:00.261731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:00.261698 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ec9799-3803-4acd-aac6-51c0f8bf1118" path="/var/lib/kubelet/pods/d9ec9799-3803-4acd-aac6-51c0f8bf1118/volumes" Oct 31 03:14:00.532782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:00.532736 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:01.313937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:01.313900 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:01.710739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:01.710702 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:02.039465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:02.039327 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:02.039639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:02.039556 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg"] Oct 31 03:14:02.045509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:02.045474 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fc4d867-sxxgg"] Oct 31 03:14:02.261869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:02.261838 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec429f23-9d6e-4960-baff-fcc4d6f548b8" path="/var/lib/kubelet/pods/ec429f23-9d6e-4960-baff-fcc4d6f548b8/volumes" Oct 31 03:14:02.710071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:02.710037 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:03.443960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:03.443923 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:04.219251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:04.219158 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:04.694144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:04.694107 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:05.002730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:05.002650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:05.799090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:05.799058 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:05.926671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:05.926623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:06.723128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:06.723096 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:07.027901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:07.027811 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5"] Oct 31 03:14:07.033199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:07.033170 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c0e3c9-tvft5"] Oct 31 03:14:07.321952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:07.321862 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:08.111256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:08.111223 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:08.263656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:08.263621 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9697a4c2-87bb-4948-ab51-ea6a383a2727" path="/var/lib/kubelet/pods/9697a4c2-87bb-4948-ab51-ea6a383a2727/volumes" Oct 31 03:14:08.433999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:08.433962 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:08.935563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:08.935531 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:09.130385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:09.130307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:09.379849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:09.379768 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:09.706430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:09.706374 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:10.428573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:10.428523 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:10.741791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:10.741695 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:14:11.170857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:11.170795 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:11.873370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:11.873331 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:12.028801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:12.028761 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:12.614784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:12.614740 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:13.518388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:13.518351 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:14.011688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:14.011655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:14.139537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:14.139501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:14.270025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:14.269939 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:14.978911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:14.978803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:15.918482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:15.918380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:15.943475 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:14:15.943441 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:14:16.535513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:16.535476 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:14:17.007460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:17.007423 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:17.425197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:17.425158 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:18.077466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:18.070822 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:19.043881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:19.043848 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:19.554016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:19.553976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:20.038107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.038047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:20.663807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.663756 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:14:20.667926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.667884 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:14:20.737737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.737689 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:14:20.888436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.867048 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:14:20.937574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.885900 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:14:20.987451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.986861 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:20.987451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.987056 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:14:20.987451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:20.987184 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:14:21.180449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.180179 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:21.193715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.193599 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.205048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.204995 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.264439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.239863 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:14:21.280936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.280879 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.300876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.299501 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.332192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.332134 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:21.349606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.349567 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:14:21.373438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.372673 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:14:21.373438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.372982 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.382250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.382220 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f1bfbe55-openshift-default-dockercfg-djzcs\"" Oct 31 03:14:21.466784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.466698 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp"] Oct 31 03:14:21.526687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.526639 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-455945c5-9lkcp"] Oct 31 03:14:21.531659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.531616 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.539605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.539567 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.540033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.540007 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.545148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.545123 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.578481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.552472 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.578783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.578759 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.579018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.578999 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.579232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.579213 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.579519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.579495 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmhz\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.680696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.680646 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:14:21.681732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.681704 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:14:21.682517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.682497 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:14:21.682699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.682685 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:14:21.684760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.684734 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.684957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.684935 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685136 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685291 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685507 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685676 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685831 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.685991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.685976 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.686155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.686135 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmhz\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.711465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.698825 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.711465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.706912 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.711465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.710111 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.712689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.712653 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.741493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.728907 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.851436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.742465 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.851436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.749357 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.851436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.760301 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:14:21.851436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.808820 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.851436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.824103 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.897245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.887150 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmhz\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:21.980509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.897490 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:21.980509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:21.898095 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:14:22.044585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.021533 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.044585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.021914 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:14:22.100965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.090893 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:22.199441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.197760 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:14:22.199441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.198516 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:14:22.199441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.198589 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:14:22.199441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.198631 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:14:22.250236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.214696 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:22.250236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.216073 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:22.250236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.216521 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:14:22.250236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.232381 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.420607 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57ab690-46ae-44fe-8db3-3cccc2b26584" path="/var/lib/kubelet/pods/c57ab690-46ae-44fe-8db3-3cccc2b26584/volumes" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421165 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421225 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421272 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421305 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421340 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421373 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:14:22.422985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.421435 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:14:22.602490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.596622 4556 scope.go:122] "RemoveContainer" containerID="6e30fc1ed9640cc2711d833f0c33751367d97f119c215ecffc33ad1e01ce5e01" Oct 31 03:14:22.631848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.629651 4556 scope.go:122] "RemoveContainer" containerID="8703caeb9626ee705750811bc18aadd9e87e46700c3b14c5333efe96bb27cec9" Oct 31 03:14:22.659067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.659041 4556 scope.go:122] "RemoveContainer" containerID="fbbbde3a2e7aa95c9b581b7ef614b178ca0c0da1a18accbd09b51cec081b47ee" Oct 31 03:14:22.705098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.705071 4556 scope.go:122] "RemoveContainer" containerID="bcb0ee5fd51b8ce3bc36c8c9459886a8effc5da414e4d27236dc25732ccc0c17" Oct 31 03:14:22.722674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.722634 4556 scope.go:122] "RemoveContainer" containerID="02d5bb494ed434c79600387f23d2ef9e73461b7461412f72f258939d941f0a93" Oct 31 03:14:22.813488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.813219 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:14:22.820903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.820873 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:22.820989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.820941 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:22.820989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:22.820971 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:23.038180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:23.033057 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:23.511594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:23.511548 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerStarted","Data":"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464"} Oct 31 03:14:23.511794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:23.511620 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerStarted","Data":"8ecb99c56b38b3e4004a849c67561aae00fb1aa3591af236539c8b7f5ca5c6fc"} Oct 31 03:14:23.539541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:23.539448 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podStartSLOduration=4.539428539 podStartE2EDuration="4.539428539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:23.534200438 +0000 UTC m=+1141.389217591" watchObservedRunningTime="2026-10-31 03:14:23.539428539 +0000 UTC m=+1141.394445672" Oct 31 03:14:23.831401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:23.831306 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:24.097895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:24.097791 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:24.104155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:24.104124 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:24.524881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:24.524662 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:24.535774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:24.535740 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:14:24.607920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:24.607853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:25.668146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:25.668110 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:26.159663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:26.159622 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:26.329057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:26.329018 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:26.738800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:26.738719 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:27.365516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:27.365394 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:27.503914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:27.503881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:27.661521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:27.648282 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:28.284697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:28.284660 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:29.100443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:29.098067 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:29.850091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:29.850043 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:30.533957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:30.533878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:31.248068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:31.246812 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:32.015177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:32.015146 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:32.815728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:32.815676 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:33.615948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:33.615891 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:34.405833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:34.405619 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:34.538603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:34.538553 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:14:34.538804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:34.538644 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:14:35.163514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.163473 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:14:35.221025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.220979 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:35.231482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.231448 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:14:35.231634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.231552 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.234768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.234740 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f1bfbe55-serving-cert\"" Oct 31 03:14:35.348637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.348601 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:35.361456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.361354 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.364465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.362031 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.364465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.362448 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.364465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.362800 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcksq\" (UniqueName: \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.464945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.464864 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.464945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.464938 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.465372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.465009 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.465372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.465108 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcksq\" (UniqueName: \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.465786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.465754 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.468144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.467934 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.468863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.468785 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.477145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.477117 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcksq\" (UniqueName: \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\") pod \"maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:35.571026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:35.570989 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:36.146045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:36.143969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:36.304368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:36.304280 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:14:36.305286 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:14:36.305254 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196598da_e0d8_4686_8ba0_069617cd19ad.slice/crio-5eb03fff6e1529b07f267d1d6303198a31885ad8f1f898a403c5ad5b8c817329 WatchSource:0}: Error finding container 5eb03fff6e1529b07f267d1d6303198a31885ad8f1f898a403c5ad5b8c817329: Status 404 returned error can't find the container with id 5eb03fff6e1529b07f267d1d6303198a31885ad8f1f898a403c5ad5b8c817329 Oct 31 03:14:36.774589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:36.774550 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" event={"ID":"196598da-e0d8-4686-8ba0-069617cd19ad","Type":"ContainerStarted","Data":"5eb03fff6e1529b07f267d1d6303198a31885ad8f1f898a403c5ad5b8c817329"} Oct 31 03:14:36.910699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:36.910650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:37.712505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.712463 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:37.786340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.786305 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" event={"ID":"196598da-e0d8-4686-8ba0-069617cd19ad","Type":"ContainerStarted","Data":"aa855de6d01c7edf4fd06a599a8416b52034c6b69bfa419eb41f951bc2172d44"} Oct 31 03:14:37.786846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.786383 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:37.788507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.788475 4556 generic.go:356] "Generic (PLEG): container finished" podID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerID="769d6fafdab299030aebc27bb6dc5bb8e8f6562186848fec51ab34cfd13c0222" exitCode=0 Oct 31 03:14:37.788642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.788524 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerDied","Data":"769d6fafdab299030aebc27bb6dc5bb8e8f6562186848fec51ab34cfd13c0222"} Oct 31 03:14:37.788958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.788941 4556 scope.go:122] "RemoveContainer" containerID="769d6fafdab299030aebc27bb6dc5bb8e8f6562186848fec51ab34cfd13c0222" Oct 31 03:14:37.790889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.790856 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:37.791002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.790944 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:37.791045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.791016 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:14:37.858398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:37.858338 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" podStartSLOduration=2.088002702 podStartE2EDuration="2.858313348s" totalImagesPullingTime="770.310646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:37.828439632 +0000 UTC m=+1155.683456775" watchObservedRunningTime="2026-10-31 03:14:37.858313348 +0000 UTC m=+1155.713330491" Oct 31 03:14:38.458367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:38.458330 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:38.811339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:38.811253 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerStarted","Data":"7935b64b2ca0ed07ad834fc46c36cd79228a1354100aa436f061e94ec16614f4"} Oct 31 03:14:39.286438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:39.281483 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:39.611972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:39.611847 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:14:39.612672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:39.612641 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:14:39.612785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:39.612700 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 03:14:39.996439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:39.994910 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:40.608484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:40.607024 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:14:40.608484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:40.607148 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 03:14:40.710804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:40.710774 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:41.451939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:41.451888 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:41.606863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:41.606834 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:14:41.607094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:41.606913 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 03:14:42.126149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:42.126113 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:42.606755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:42.606716 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:14:42.607166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:42.606775 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 03:14:42.948379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:42.942877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:43.606698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:43.606663 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:14:43.606883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:43.606718 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 31 03:14:43.624509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:43.624471 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:43.823435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:43.820673 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:14:44.431883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:44.431843 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:44.611845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:44.611814 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:14:44.612392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:44.612369 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:14:44.613176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:44.613154 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:14:45.238983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:45.238946 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:46.115150 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:14:46.115098 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:14:46.140293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:46.140258 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:47.001089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:47.001050 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:47.721874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:47.721839 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:48.524993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:48.524833 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:49.233573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:49.233532 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:49.935529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:49.935478 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:50.729288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:50.729252 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:51.533065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:51.533021 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:51.662472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:51.662387 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:52.538406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:52.538332 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:53.421793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:53.421750 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:53.768811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:53.762894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:54.523456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:54.523353 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:55.029275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:55.029236 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m"] Oct 31 03:14:55.037581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:55.037549 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f869c9-w2c9m"] Oct 31 03:14:55.422136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:55.422104 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:56.199132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:56.199086 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:56.261978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:56.261939 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2804be-3e95-4a89-9651-2e6eca65150a" path="/var/lib/kubelet/pods/4b2804be-3e95-4a89-9651-2e6eca65150a/volumes" Oct 31 03:14:57.035260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:57.035225 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:57.817104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:57.817062 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:58.030808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:58.030765 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc"] Oct 31 03:14:58.036208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:58.036177 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51706f70-sbprc"] Oct 31 03:14:58.263046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:58.263013 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53de3723-ad06-4f28-992a-6dcbffeac454" path="/var/lib/kubelet/pods/53de3723-ad06-4f28-992a-6dcbffeac454/volumes" Oct 31 03:14:58.619491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:58.619263 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:14:59.438365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:14:59.438336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:00.145133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.145074 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:00.175933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.175735 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:00.233171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.233137 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:00.236459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.234989 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.269488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.262263 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:00.282680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.268403 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.296748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.296716 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:15:00.300676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.299315 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.326811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.326780 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:00.334808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.334771 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:00.335184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.335104 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.336596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.336577 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:00.336783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.336765 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:00.337301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.337283 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:00.347820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.347785 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:00.347944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.347836 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:00.347990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.347943 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.361764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.361738 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbxj\" (UniqueName: \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.361908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.361795 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.361963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.361936 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9szf\" (UniqueName: \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.362103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.362082 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.463821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.463780 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.464299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.463838 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbxj\" (UniqueName: \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.464299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.463933 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.464299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.463992 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.464299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464084 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9szf\" (UniqueName: \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.464299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464249 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.464624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464326 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnfx\" (UniqueName: \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.464624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464392 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.464624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464474 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prrgk\" (UniqueName: \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.464624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464533 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9hz\" (UniqueName: \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.464903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464881 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.464963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.464940 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.477944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.477919 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbxj\" (UniqueName: \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\") pod \"maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.478068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.477917 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9szf\" (UniqueName: \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\") pod \"maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.565928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.565891 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.566104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566091 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnfx\" (UniqueName: \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.566159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566132 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.566159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566156 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrgk\" (UniqueName: \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.566234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566178 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9hz\" (UniqueName: \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.566282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566266 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.566778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566754 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.566893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566873 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.566942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.566904 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.577720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.577694 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrgk\" (UniqueName: \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\") pod \"maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.578039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.578012 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnfx\" (UniqueName: \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\") pod \"maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.579402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.579379 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9hz\" (UniqueName: \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\") pod \"maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.599215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.599191 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:00.619983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.619937 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:00.635328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.635303 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:00.671328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.670626 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:00.678662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.678056 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:00.896097 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:00.896055 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8df3b11_da38_4e18_8218_17555a883aaf.slice/crio-2b1c9483f54f839ca5f66503f4cf643301c208d55d5c6a82e959aa8d6c67a142 WatchSource:0}: Error finding container 2b1c9483f54f839ca5f66503f4cf643301c208d55d5c6a82e959aa8d6c67a142: Status 404 returned error can't find the container with id 2b1c9483f54f839ca5f66503f4cf643301c208d55d5c6a82e959aa8d6c67a142 Oct 31 03:15:00.900527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.900501 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:00.942389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.942357 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:00.955338 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:00.955295 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85538fce_66f3_4dc8_b12e_791678523337.slice/crio-da65ea455f4642be1db56a8841287bab474635e8384526ad78c4f4c0676a349e WatchSource:0}: Error finding container da65ea455f4642be1db56a8841287bab474635e8384526ad78c4f4c0676a349e: Status 404 returned error can't find the container with id da65ea455f4642be1db56a8841287bab474635e8384526ad78c4f4c0676a349e Oct 31 03:15:00.972865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:00.972086 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:01.022589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.022563 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:01.028616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.028589 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:01.070550 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:01.070517 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab481662_5ab8_4a37_979f_2692e1595ab3.slice/crio-fe742000bafa8ce7440c835b247a881bf8bda2c3ee1cc419020b0bf141566fd6 WatchSource:0}: Error finding container fe742000bafa8ce7440c835b247a881bf8bda2c3ee1cc419020b0bf141566fd6: Status 404 returned error can't find the container with id fe742000bafa8ce7440c835b247a881bf8bda2c3ee1cc419020b0bf141566fd6 Oct 31 03:15:01.072335 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:01.072305 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ab58e5_f40f_4d51_b585_cde97b0799e2.slice/crio-313bfc564d93d515130c3e96da59e56127f92612b7811ebeb87e64b5048b3d5a WatchSource:0}: Error finding container 313bfc564d93d515130c3e96da59e56127f92612b7811ebeb87e64b5048b3d5a: Status 404 returned error can't find the container with id 313bfc564d93d515130c3e96da59e56127f92612b7811ebeb87e64b5048b3d5a Oct 31 03:15:01.073303 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:01.073252 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c274a10_83bb_406a_a3b8_4a358563d5f0.slice/crio-83bdb4f392368eccd199ed5d187ae42719fec26f7431cb4dbceb4b584c79d54e WatchSource:0}: Error finding container 83bdb4f392368eccd199ed5d187ae42719fec26f7431cb4dbceb4b584c79d54e: Status 404 returned error can't find the container with id 83bdb4f392368eccd199ed5d187ae42719fec26f7431cb4dbceb4b584c79d54e Oct 31 03:15:01.220258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.220201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:01.253590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.253495 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerStarted","Data":"fe742000bafa8ce7440c835b247a881bf8bda2c3ee1cc419020b0bf141566fd6"} Oct 31 03:15:01.257345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.257311 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerStarted","Data":"2b1c9483f54f839ca5f66503f4cf643301c208d55d5c6a82e959aa8d6c67a142"} Oct 31 03:15:01.260750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.260718 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerStarted","Data":"83bdb4f392368eccd199ed5d187ae42719fec26f7431cb4dbceb4b584c79d54e"} Oct 31 03:15:01.264536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.264497 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" event={"ID":"f3ab58e5-f40f-4d51-b585-cde97b0799e2","Type":"ContainerStarted","Data":"313bfc564d93d515130c3e96da59e56127f92612b7811ebeb87e64b5048b3d5a"} Oct 31 03:15:01.271620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:01.271576 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerStarted","Data":"da65ea455f4642be1db56a8841287bab474635e8384526ad78c4f4c0676a349e"} Oct 31 03:15:02.221847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.221165 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:02.303745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.303703 4556 generic.go:356] "Generic (PLEG): container finished" podID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerID="0da9ad2288e4a47aaa0cb98979e747255ca22bfa4b2e55642a2e785ae6b6ac95" exitCode=60 Oct 31 03:15:02.303905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.303784 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" event={"ID":"f3ab58e5-f40f-4d51-b585-cde97b0799e2","Type":"ContainerDied","Data":"0da9ad2288e4a47aaa0cb98979e747255ca22bfa4b2e55642a2e785ae6b6ac95"} Oct 31 03:15:02.304073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.304054 4556 scope.go:122] "RemoveContainer" containerID="0da9ad2288e4a47aaa0cb98979e747255ca22bfa4b2e55642a2e785ae6b6ac95" Oct 31 03:15:02.346589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.311967 4556 generic.go:356] "Generic (PLEG): container finished" podID="85538fce-66f3-4dc8-b12e-791678523337" containerID="cbcdbf72f996d1f63a31f3710dc1931e5d8e43e3bc8ebbeaaa7b2f1bcb630062" exitCode=60 Oct 31 03:15:02.346589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.312611 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerDied","Data":"cbcdbf72f996d1f63a31f3710dc1931e5d8e43e3bc8ebbeaaa7b2f1bcb630062"} Oct 31 03:15:02.346589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.312826 4556 scope.go:122] "RemoveContainer" containerID="cbcdbf72f996d1f63a31f3710dc1931e5d8e43e3bc8ebbeaaa7b2f1bcb630062" Oct 31 03:15:02.370731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.370241 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerStarted","Data":"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b"} Oct 31 03:15:02.370989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.370778 4556 scope.go:122] "RemoveContainer" containerID="d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b" Oct 31 03:15:02.381244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.380983 4556 generic.go:356] "Generic (PLEG): container finished" podID="a8df3b11-da38-4e18-8218-17555a883aaf" containerID="ba0b257cb4f25714799ea3f2b1b35a194020ff6249efe78d04438a87274369d4" exitCode=60 Oct 31 03:15:02.381566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.381543 4556 scope.go:122] "RemoveContainer" containerID="ba0b257cb4f25714799ea3f2b1b35a194020ff6249efe78d04438a87274369d4" Oct 31 03:15:02.382037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.381117 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerDied","Data":"ba0b257cb4f25714799ea3f2b1b35a194020ff6249efe78d04438a87274369d4"} Oct 31 03:15:02.387787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.387760 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerStarted","Data":"42a439b6eec4d017a171d1c968ddbca0930b52f99e7c2158df1f28acc7e44049"} Oct 31 03:15:02.432012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:02.431963 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" podStartSLOduration=2.431946904 podStartE2EDuration="2.431946904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:02.424774012 +0000 UTC m=+1180.279791155" watchObservedRunningTime="2026-10-31 03:15:02.431946904 +0000 UTC m=+1180.286964071" Oct 31 03:15:03.131327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.131218 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:03.398734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.398652 4556 generic.go:356] "Generic (PLEG): container finished" podID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerID="42a439b6eec4d017a171d1c968ddbca0930b52f99e7c2158df1f28acc7e44049" exitCode=60 Oct 31 03:15:03.399161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.398737 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerDied","Data":"42a439b6eec4d017a171d1c968ddbca0930b52f99e7c2158df1f28acc7e44049"} Oct 31 03:15:03.399161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.399058 4556 scope.go:122] "RemoveContainer" containerID="42a439b6eec4d017a171d1c968ddbca0930b52f99e7c2158df1f28acc7e44049" Oct 31 03:15:03.401297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.401270 4556 generic.go:356] "Generic (PLEG): container finished" podID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerID="eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4" exitCode=60 Oct 31 03:15:03.401452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.401359 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" event={"ID":"f3ab58e5-f40f-4d51-b585-cde97b0799e2","Type":"ContainerDied","Data":"eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4"} Oct 31 03:15:03.401452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.401431 4556 scope.go:122] "RemoveContainer" containerID="0da9ad2288e4a47aaa0cb98979e747255ca22bfa4b2e55642a2e785ae6b6ac95" Oct 31 03:15:03.401734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.401713 4556 scope.go:122] "RemoveContainer" containerID="eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4" Oct 31 03:15:03.402191 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:03.402168 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs_odh-ai-gateway-infra(f3ab58e5-f40f-4d51-b585-cde97b0799e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" Oct 31 03:15:03.403913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.403890 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerStarted","Data":"14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796"} Oct 31 03:15:03.406075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.406050 4556 generic.go:356] "Generic (PLEG): container finished" podID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerID="d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b" exitCode=60 Oct 31 03:15:03.406075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.406074 4556 generic.go:356] "Generic (PLEG): container finished" podID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerID="a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d" exitCode=60 Oct 31 03:15:03.406230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.406131 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerDied","Data":"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b"} Oct 31 03:15:03.406230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.406186 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerDied","Data":"a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d"} Oct 31 03:15:03.406394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.406376 4556 scope.go:122] "RemoveContainer" containerID="a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d" Oct 31 03:15:03.406722 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:03.406693 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k_odh-ai-gateway-infra(ab481662-5ab8-4a37-979f-2692e1595ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" Oct 31 03:15:03.409848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.409824 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerStarted","Data":"1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117"} Oct 31 03:15:03.453969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.453913 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" podStartSLOduration=3.453896361 podStartE2EDuration="3.453896361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.449860295 +0000 UTC m=+1181.304877436" watchObservedRunningTime="2026-10-31 03:15:03.453896361 +0000 UTC m=+1181.308913504" Oct 31 03:15:03.471237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.471187 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" podStartSLOduration=3.471171989 podStartE2EDuration="3.471171989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.465799424 +0000 UTC m=+1181.320816564" watchObservedRunningTime="2026-10-31 03:15:03.471171989 +0000 UTC m=+1181.326189167" Oct 31 03:15:03.638733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.638700 4556 scope.go:122] "RemoveContainer" containerID="d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b" Oct 31 03:15:03.654326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.654300 4556 scope.go:122] "RemoveContainer" containerID="d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b" Oct 31 03:15:03.654681 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:03.654659 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b\": container with ID starting with d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b not found: ID does not exist" containerID="d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b" Oct 31 03:15:03.654758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:03.654690 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b"} err="failed to get container status \"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b\": rpc error: code = NotFound desc = could not find container \"d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b\": container with ID starting with d57f0f2e8b4141575df532d7b3fda6c45d9f00723ed9bc6479d114f877984f6b not found: ID does not exist" Oct 31 03:15:04.138743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.138707 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:04.424730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.424684 4556 scope.go:122] "RemoveContainer" containerID="a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d" Oct 31 03:15:04.425228 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:04.425176 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k_odh-ai-gateway-infra(ab481662-5ab8-4a37-979f-2692e1595ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" Oct 31 03:15:04.426719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.426694 4556 generic.go:356] "Generic (PLEG): container finished" podID="a8df3b11-da38-4e18-8218-17555a883aaf" containerID="1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117" exitCode=60 Oct 31 03:15:04.426835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.426769 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerDied","Data":"1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117"} Oct 31 03:15:04.426835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.426829 4556 scope.go:122] "RemoveContainer" containerID="ba0b257cb4f25714799ea3f2b1b35a194020ff6249efe78d04438a87274369d4" Oct 31 03:15:04.427121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.427103 4556 scope.go:122] "RemoveContainer" containerID="1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117" Oct 31 03:15:04.427545 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:04.427522 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw_odh-ai-gateway-infra(a8df3b11-da38-4e18-8218-17555a883aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" Oct 31 03:15:04.430672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.430646 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerStarted","Data":"92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26"} Oct 31 03:15:04.433332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.433309 4556 scope.go:122] "RemoveContainer" containerID="eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4" Oct 31 03:15:04.433730 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:04.433707 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs_odh-ai-gateway-infra(f3ab58e5-f40f-4d51-b585-cde97b0799e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" Oct 31 03:15:04.435113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.435090 4556 generic.go:356] "Generic (PLEG): container finished" podID="85538fce-66f3-4dc8-b12e-791678523337" containerID="14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796" exitCode=60 Oct 31 03:15:04.435220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.435150 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerDied","Data":"14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796"} Oct 31 03:15:04.435397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.435382 4556 scope.go:122] "RemoveContainer" containerID="14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796" Oct 31 03:15:04.435718 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:04.435690 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb_odh-ai-gateway-infra(85538fce-66f3-4dc8-b12e-791678523337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" podUID="85538fce-66f3-4dc8-b12e-791678523337" Oct 31 03:15:04.445186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.445155 4556 scope.go:122] "RemoveContainer" containerID="cbcdbf72f996d1f63a31f3710dc1931e5d8e43e3bc8ebbeaaa7b2f1bcb630062" Oct 31 03:15:04.900277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:04.900234 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:05.446844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.446812 4556 scope.go:122] "RemoveContainer" containerID="1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117" Oct 31 03:15:05.447260 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:05.447241 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw_odh-ai-gateway-infra(a8df3b11-da38-4e18-8218-17555a883aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" Oct 31 03:15:05.448698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.448672 4556 generic.go:356] "Generic (PLEG): container finished" podID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerID="92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26" exitCode=60 Oct 31 03:15:05.448798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.448737 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerDied","Data":"92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26"} Oct 31 03:15:05.448839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.448804 4556 scope.go:122] "RemoveContainer" containerID="42a439b6eec4d017a171d1c968ddbca0930b52f99e7c2158df1f28acc7e44049" Oct 31 03:15:05.449088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.449068 4556 scope.go:122] "RemoveContainer" containerID="92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26" Oct 31 03:15:05.449513 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:05.449486 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64_odh-ai-gateway-infra(1c274a10-83bb-406a-a3b8-4a358563d5f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" Oct 31 03:15:05.451184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.451167 4556 scope.go:122] "RemoveContainer" containerID="14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796" Oct 31 03:15:05.451487 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:05.451468 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb_odh-ai-gateway-infra(85538fce-66f3-4dc8-b12e-791678523337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" podUID="85538fce-66f3-4dc8-b12e-791678523337" Oct 31 03:15:05.756065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:05.755976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:06.471812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:06.471775 4556 scope.go:122] "RemoveContainer" containerID="92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26" Oct 31 03:15:06.472499 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:06.472461 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64_odh-ai-gateway-infra(1c274a10-83bb-406a-a3b8-4a358563d5f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" Oct 31 03:15:06.626969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:06.626695 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:07.319355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:07.318467 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:08.010138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.010057 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:08.053361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.053309 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs"] Oct 31 03:15:08.063261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.063224 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dfe12f1-d6jqs"] Oct 31 03:15:08.260841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.260761 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190ac8ad-aade-4896-a39c-4a5a6ca93091" path="/var/lib/kubelet/pods/190ac8ad-aade-4896-a39c-4a5a6ca93091/volumes" Oct 31 03:15:08.430041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.430010 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:08.833449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:08.831256 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:09.029122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:09.029086 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497"] Oct 31 03:15:09.034823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:09.034790 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c8e562-4t497"] Oct 31 03:15:09.430235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:09.430200 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:09.624931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:09.624892 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:10.263194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:10.263156 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a249c8-7212-4871-b5be-84e14e714f74" path="/var/lib/kubelet/pods/04a249c8-7212-4871-b5be-84e14e714f74/volumes" Oct 31 03:15:10.460879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:10.460825 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:11.227457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:11.225806 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:11.928083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:11.928038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:12.630514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:12.630482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:13.332588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:13.332548 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:14.228474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:14.228430 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:14.931508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:14.931462 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:15.250738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:15.250652 4556 scope.go:122] "RemoveContainer" containerID="14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796" Oct 31 03:15:15.937369 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:15.937279 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:15:16.251010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:16.250927 4556 scope.go:122] "RemoveContainer" containerID="1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117" Oct 31 03:15:16.583290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:16.583211 4556 generic.go:356] "Generic (PLEG): container finished" podID="85538fce-66f3-4dc8-b12e-791678523337" containerID="584fff8eb3356d83f08ac3351a3749a96dd24a2bad07ac0bdfe1186415ae4e72" exitCode=60 Oct 31 03:15:16.583460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:16.583288 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerDied","Data":"584fff8eb3356d83f08ac3351a3749a96dd24a2bad07ac0bdfe1186415ae4e72"} Oct 31 03:15:16.583460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:16.583360 4556 scope.go:122] "RemoveContainer" containerID="14a86568a502650173c28ab3f65472b1fb9b7991e06dc64105e421758dad2796" Oct 31 03:15:16.583897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:16.583872 4556 scope.go:122] "RemoveContainer" containerID="584fff8eb3356d83f08ac3351a3749a96dd24a2bad07ac0bdfe1186415ae4e72" Oct 31 03:15:16.584374 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:16.584344 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb_odh-ai-gateway-infra(85538fce-66f3-4dc8-b12e-791678523337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" podUID="85538fce-66f3-4dc8-b12e-791678523337" Oct 31 03:15:17.251218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.251174 4556 scope.go:122] "RemoveContainer" containerID="a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d" Oct 31 03:15:17.598696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.598660 4556 generic.go:356] "Generic (PLEG): container finished" podID="a8df3b11-da38-4e18-8218-17555a883aaf" containerID="f993611fcbd5a000ebb253c580bdb56b37032fabaad6df574e755fe4b8e672e1" exitCode=60 Oct 31 03:15:17.598862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.598725 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerDied","Data":"f993611fcbd5a000ebb253c580bdb56b37032fabaad6df574e755fe4b8e672e1"} Oct 31 03:15:17.598862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.598784 4556 scope.go:122] "RemoveContainer" containerID="1aff7e0e83d4c6470f9ab45a5176427c12f30d3405041676a4bbe3c7bb1f0117" Oct 31 03:15:17.599220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.599202 4556 scope.go:122] "RemoveContainer" containerID="f993611fcbd5a000ebb253c580bdb56b37032fabaad6df574e755fe4b8e672e1" Oct 31 03:15:17.599689 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:17.599664 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw_odh-ai-gateway-infra(a8df3b11-da38-4e18-8218-17555a883aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" Oct 31 03:15:17.617972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.617921 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:17.791338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.791311 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:17.899166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.899088 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\" (UniqueName: \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\") pod \"85538fce-66f3-4dc8-b12e-791678523337\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " Oct 31 03:15:17.899298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.899278 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\") pod \"85538fce-66f3-4dc8-b12e-791678523337\" (UID: \"85538fce-66f3-4dc8-b12e-791678523337\") " Oct 31 03:15:17.899618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.899588 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca" pod "85538fce-66f3-4dc8-b12e-791678523337" (UID: "85538fce-66f3-4dc8-b12e-791678523337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:17.899930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.899903 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85538fce-66f3-4dc8-b12e-791678523337-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:17.903268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:17.903245 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj" pod "85538fce-66f3-4dc8-b12e-791678523337" (UID: "85538fce-66f3-4dc8-b12e-791678523337"). InnerVolumeSpecName "kube-api-access-bdbxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:18.002162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.002128 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdbxj\" (UniqueName: \"kubernetes.io/projected/85538fce-66f3-4dc8-b12e-791678523337-kube-api-access-bdbxj\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.250643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.250609 4556 scope.go:122] "RemoveContainer" containerID="92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26" Oct 31 03:15:18.250850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.250782 4556 scope.go:122] "RemoveContainer" containerID="eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4" Oct 31 03:15:18.613033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.612999 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" Oct 31 03:15:18.613571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.613034 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb" event={"ID":"85538fce-66f3-4dc8-b12e-791678523337","Type":"ContainerDied","Data":"da65ea455f4642be1db56a8841287bab474635e8384526ad78c4f4c0676a349e"} Oct 31 03:15:18.613571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.613111 4556 scope.go:122] "RemoveContainer" containerID="584fff8eb3356d83f08ac3351a3749a96dd24a2bad07ac0bdfe1186415ae4e72" Oct 31 03:15:18.615973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.615941 4556 generic.go:356] "Generic (PLEG): container finished" podID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerID="ee0ddb58bd98e35125f04a73d7b50c0d54f9e947f673ed045e49b9a636d8c180" exitCode=60 Oct 31 03:15:18.616087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.616011 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerDied","Data":"ee0ddb58bd98e35125f04a73d7b50c0d54f9e947f673ed045e49b9a636d8c180"} Oct 31 03:15:18.616787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.616765 4556 scope.go:122] "RemoveContainer" containerID="ee0ddb58bd98e35125f04a73d7b50c0d54f9e947f673ed045e49b9a636d8c180" Oct 31 03:15:18.617159 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:18.617136 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k_odh-ai-gateway-infra(ab481662-5ab8-4a37-979f-2692e1595ab3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" Oct 31 03:15:18.628649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.628633 4556 scope.go:122] "RemoveContainer" containerID="a3d25b64a8bd0570519fb255e67640e2d811a703fba0cc6c1fa431b67364f43d" Oct 31 03:15:18.630204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.630187 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:18.651338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.651310 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:18.662649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.662621 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f1bfbe55-29890275-qqmrb"] Oct 31 03:15:18.846744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.846720 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:18.918227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.918176 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\") pod \"a8df3b11-da38-4e18-8218-17555a883aaf\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " Oct 31 03:15:18.918388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.918303 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\" (UniqueName: \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\") pod \"a8df3b11-da38-4e18-8218-17555a883aaf\" (UID: \"a8df3b11-da38-4e18-8218-17555a883aaf\") " Oct 31 03:15:18.918652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.918620 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca" pod "a8df3b11-da38-4e18-8218-17555a883aaf" (UID: "a8df3b11-da38-4e18-8218-17555a883aaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:18.920105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.919612 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df3b11-da38-4e18-8218-17555a883aaf-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:18.922168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:18.922139 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf" pod "a8df3b11-da38-4e18-8218-17555a883aaf" (UID: "a8df3b11-da38-4e18-8218-17555a883aaf"). InnerVolumeSpecName "kube-api-access-x9szf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:19.022438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.022382 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9szf\" (UniqueName: \"kubernetes.io/projected/a8df3b11-da38-4e18-8218-17555a883aaf-kube-api-access-x9szf\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:19.629425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.629384 4556 generic.go:356] "Generic (PLEG): container finished" podID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerID="187317322e2008171f043cb1d104e513e3c9486e9efa0eace20508df3f562e25" exitCode=60 Oct 31 03:15:19.629861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.629457 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerDied","Data":"187317322e2008171f043cb1d104e513e3c9486e9efa0eace20508df3f562e25"} Oct 31 03:15:19.629861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.629516 4556 scope.go:122] "RemoveContainer" containerID="92147abeeb0b19e1a5f417cffc532c9f0887b3352a85111b710d88e58c6d3c26" Oct 31 03:15:19.630052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.630028 4556 scope.go:122] "RemoveContainer" containerID="187317322e2008171f043cb1d104e513e3c9486e9efa0eace20508df3f562e25" Oct 31 03:15:19.630781 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:19.630508 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64_odh-ai-gateway-infra(1c274a10-83bb-406a-a3b8-4a358563d5f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" Oct 31 03:15:19.631946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.631913 4556 generic.go:356] "Generic (PLEG): container finished" podID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerID="88da7fbd120497fecdfb4ebf18e9d9e00c475e801c513a0974530b08ba9c3de4" exitCode=60 Oct 31 03:15:19.632031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.631983 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" event={"ID":"f3ab58e5-f40f-4d51-b585-cde97b0799e2","Type":"ContainerDied","Data":"88da7fbd120497fecdfb4ebf18e9d9e00c475e801c513a0974530b08ba9c3de4"} Oct 31 03:15:19.632363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.632347 4556 scope.go:122] "RemoveContainer" containerID="88da7fbd120497fecdfb4ebf18e9d9e00c475e801c513a0974530b08ba9c3de4" Oct 31 03:15:19.632746 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:19.632725 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs_odh-ai-gateway-infra(f3ab58e5-f40f-4d51-b585-cde97b0799e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" Oct 31 03:15:19.636443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.636390 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" event={"ID":"a8df3b11-da38-4e18-8218-17555a883aaf","Type":"ContainerDied","Data":"2b1c9483f54f839ca5f66503f4cf643301c208d55d5c6a82e959aa8d6c67a142"} Oct 31 03:15:19.636579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.636402 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw" Oct 31 03:15:19.651756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.651731 4556 scope.go:122] "RemoveContainer" containerID="eac54deff590a8f7bba9137fae2c253f4f95a3f059fea74ff710a34dd7560bf4" Oct 31 03:15:19.658530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.658499 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:19.670304 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.670271 4556 scope.go:122] "RemoveContainer" containerID="f993611fcbd5a000ebb253c580bdb56b37032fabaad6df574e755fe4b8e672e1" Oct 31 03:15:19.689950 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.689921 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:19.696792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.696754 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-edb9e6-29890275-k9pkw"] Oct 31 03:15:19.887095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.887070 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:19.939456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.939373 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\" (UniqueName: \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\") pod \"ab481662-5ab8-4a37-979f-2692e1595ab3\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " Oct 31 03:15:19.939785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.939761 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\") pod \"ab481662-5ab8-4a37-979f-2692e1595ab3\" (UID: \"ab481662-5ab8-4a37-979f-2692e1595ab3\") " Oct 31 03:15:19.940123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.940102 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca" pod "ab481662-5ab8-4a37-979f-2692e1595ab3" (UID: "ab481662-5ab8-4a37-979f-2692e1595ab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:19.940678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.940654 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab481662-5ab8-4a37-979f-2692e1595ab3-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:19.941543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:19.941523 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk" pod "ab481662-5ab8-4a37-979f-2692e1595ab3" (UID: "ab481662-5ab8-4a37-979f-2692e1595ab3"). InnerVolumeSpecName "kube-api-access-prrgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:20.042281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.042243 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prrgk\" (UniqueName: \"kubernetes.io/projected/ab481662-5ab8-4a37-979f-2692e1595ab3-kube-api-access-prrgk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.262602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.262566 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85538fce-66f3-4dc8-b12e-791678523337" path="/var/lib/kubelet/pods/85538fce-66f3-4dc8-b12e-791678523337/volumes" Oct 31 03:15:20.263250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.263233 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" path="/var/lib/kubelet/pods/a8df3b11-da38-4e18-8218-17555a883aaf/volumes" Oct 31 03:15:20.264526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.264496 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:20.531538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.529992 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:20.655731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.655692 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" Oct 31 03:15:20.656196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.655697 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k" event={"ID":"ab481662-5ab8-4a37-979f-2692e1595ab3","Type":"ContainerDied","Data":"fe742000bafa8ce7440c835b247a881bf8bda2c3ee1cc419020b0bf141566fd6"} Oct 31 03:15:20.656196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.655840 4556 scope.go:122] "RemoveContainer" containerID="ee0ddb58bd98e35125f04a73d7b50c0d54f9e947f673ed045e49b9a636d8c180" Oct 31 03:15:20.669667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.669638 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:20.686778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.686747 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:20.692373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.692345 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:20.706335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.704257 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-99bc97-29890275-2hn5k"] Oct 31 03:15:20.867287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.867258 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:20.922248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.922225 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:20.959353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.959318 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\") pod \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " Oct 31 03:15:20.959643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.959624 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\" (UniqueName: \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\") pod \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\" (UID: \"f3ab58e5-f40f-4d51-b585-cde97b0799e2\") " Oct 31 03:15:20.959736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.959657 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca" pod "f3ab58e5-f40f-4d51-b585-cde97b0799e2" (UID: "f3ab58e5-f40f-4d51-b585-cde97b0799e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:20.959790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.959745 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\" (UniqueName: \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\") pod \"1c274a10-83bb-406a-a3b8-4a358563d5f0\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " Oct 31 03:15:20.959845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.959825 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\") pod \"1c274a10-83bb-406a-a3b8-4a358563d5f0\" (UID: \"1c274a10-83bb-406a-a3b8-4a358563d5f0\") " Oct 31 03:15:20.960637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.960178 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca" pod "1c274a10-83bb-406a-a3b8-4a358563d5f0" (UID: "1c274a10-83bb-406a-a3b8-4a358563d5f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:20.960637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.960601 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c274a10-83bb-406a-a3b8-4a358563d5f0-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.960637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.960619 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ab58e5-f40f-4d51-b585-cde97b0799e2-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:20.961879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.961850 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx" pod "f3ab58e5-f40f-4d51-b585-cde97b0799e2" (UID: "f3ab58e5-f40f-4d51-b585-cde97b0799e2"). InnerVolumeSpecName "kube-api-access-fqnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:20.962483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:20.962386 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz" pod "1c274a10-83bb-406a-a3b8-4a358563d5f0" (UID: "1c274a10-83bb-406a-a3b8-4a358563d5f0"). InnerVolumeSpecName "kube-api-access-ws9hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:21.029886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.029854 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8"] Oct 31 03:15:21.035432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.035359 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da27fc3a-f2jp8"] Oct 31 03:15:21.062591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.062559 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnfx\" (UniqueName: \"kubernetes.io/projected/f3ab58e5-f40f-4d51-b585-cde97b0799e2-kube-api-access-fqnfx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:21.062591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.062588 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws9hz\" (UniqueName: \"kubernetes.io/projected/1c274a10-83bb-406a-a3b8-4a358563d5f0-kube-api-access-ws9hz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:21.314007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.313902 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.672816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.672786 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" Oct 31 03:15:21.673254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.672819 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs" event={"ID":"f3ab58e5-f40f-4d51-b585-cde97b0799e2","Type":"ContainerDied","Data":"313bfc564d93d515130c3e96da59e56127f92612b7811ebeb87e64b5048b3d5a"} Oct 31 03:15:21.673254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.672892 4556 scope.go:122] "RemoveContainer" containerID="88da7fbd120497fecdfb4ebf18e9d9e00c475e801c513a0974530b08ba9c3de4" Oct 31 03:15:21.676100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.676077 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" Oct 31 03:15:21.676225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.676112 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64" event={"ID":"1c274a10-83bb-406a-a3b8-4a358563d5f0","Type":"ContainerDied","Data":"83bdb4f392368eccd199ed5d187ae42719fec26f7431cb4dbceb4b584c79d54e"} Oct 31 03:15:21.687022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.687004 4556 scope.go:122] "RemoveContainer" containerID="187317322e2008171f043cb1d104e513e3c9486e9efa0eace20508df3f562e25" Oct 31 03:15:21.700899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.700876 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:21.709076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.709049 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-924ee5-29890275-jt8cs"] Oct 31 03:15:21.731937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.731888 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:21.735969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:21.735941 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-85cb59-29890275-fxs64"] Oct 31 03:15:22.044897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.044757 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:22.260467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.260432 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007eeebc-f8cf-428c-aee1-aec8757a1271" path="/var/lib/kubelet/pods/007eeebc-f8cf-428c-aee1-aec8757a1271/volumes" Oct 31 03:15:22.260922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.260904 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" path="/var/lib/kubelet/pods/1c274a10-83bb-406a-a3b8-4a358563d5f0/volumes" Oct 31 03:15:22.261373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.261345 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" path="/var/lib/kubelet/pods/ab481662-5ab8-4a37-979f-2692e1595ab3/volumes" Oct 31 03:15:22.262230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.262210 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" path="/var/lib/kubelet/pods/f3ab58e5-f40f-4d51-b585-cde97b0799e2/volumes" Oct 31 03:15:22.824485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.824448 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:22.868798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.868763 4556 scope.go:122] "RemoveContainer" containerID="3814d79e80fe0274e5053403bbd3808e37d71e31a37a9f4e1616536c8006c8da" Oct 31 03:15:22.881381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.881354 4556 scope.go:122] "RemoveContainer" containerID="481df0f950b81d265e784b81b2b429e0c0511809a360d14a5454e69eea77b4f2" Oct 31 03:15:22.894023 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.893996 4556 scope.go:122] "RemoveContainer" containerID="b5d2c17bcde29d0c4c6c91d67f05d906975236ffbe05e6c18c757821d302910a" Oct 31 03:15:22.910705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.910195 4556 scope.go:122] "RemoveContainer" containerID="20183d1a6f0f7bdfe1681a4e26394c0b05b53002a2cc0f3d333f506b1c0b885e" Oct 31 03:15:22.927929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.927905 4556 scope.go:122] "RemoveContainer" containerID="8a08a4bf42e7958fb91b15989189d9a8bd098f87292ccb70c14d8e0069c98fb8" Oct 31 03:15:22.944977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.944956 4556 scope.go:122] "RemoveContainer" containerID="af7add7329096340a632a8ca59ed6185361bdae97d3197268366593dadb3cbbf" Oct 31 03:15:22.960515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.960480 4556 scope.go:122] "RemoveContainer" containerID="5391ad3c08eea3b7ad7f38d1cf4c826d60e4e7d4663dcfc2fef674f96e5c461e" Oct 31 03:15:22.974701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:22.974677 4556 scope.go:122] "RemoveContainer" containerID="51225e995d1efe04404857db0338f71a422970354f0e4289b0ea588944109cf8" Oct 31 03:15:23.011650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.011628 4556 scope.go:122] "RemoveContainer" containerID="8bae400bd272125d7ac8e73f6cba35c9634beb36538fdbcbce6dbeef495d8c39" Oct 31 03:15:23.022801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.022778 4556 scope.go:122] "RemoveContainer" containerID="5b884206185b10153326c3c807300399f1f22693ce38e45d00b05bb32329d58a" Oct 31 03:15:23.033403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.033380 4556 scope.go:122] "RemoveContainer" containerID="c54e3ebfaae749e9be5bea14f49c5b7841f3bb6c9d4360be7be132ea32a6f94c" Oct 31 03:15:23.043691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.043663 4556 scope.go:122] "RemoveContainer" containerID="e2837e04b46a80b06f6dba17d9a6690d48267c42917dd46459152de4421162af" Oct 31 03:15:23.053763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.053742 4556 scope.go:122] "RemoveContainer" containerID="d8acc0de0fe30040a46a39f86936189504aab77615da25c994c660b52fe104d7" Oct 31 03:15:23.518089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:23.518038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.209311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:24.209279 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.757207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:24.757170 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.980268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:24.980224 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.115135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.115051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.264289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.264253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.346160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.346118 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:15:25.379131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379054 4556 cpu_manager.go:414] "containerMap: removing container" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379083 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379102 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379110 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379134 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379139 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379150 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379155 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379161 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379166 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379172 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379176 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379195 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379201 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379209 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379213 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379228 4556 cpu_manager.go:414] "containerMap: removing container" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379232 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379240 4556 cpu_manager.go:414] "containerMap: removing container" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379244 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379253 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379257 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379274 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379280 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379427 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379441 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379450 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379458 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379464 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379473 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379479 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379487 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379495 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="85538fce-66f3-4dc8-b12e-791678523337" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379504 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379511 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:25.379521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.379516 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:25.387494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.387465 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:15:25.387639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.387543 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.391465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.390734 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2206e7-openshift-default-dockercfg-t49h2\"" Oct 31 03:15:25.419097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.419066 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.503907 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504030 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504119 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504227 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504292 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcpf\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504325 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504504 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504538 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.509435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.504649 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.551435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.550939 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.606617 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.606760 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.606839 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.606913 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.607003 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.607059 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcpf\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.607088 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.607218 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.607257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.607248 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.653488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.623745 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.653488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.624303 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.653488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.628801 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.653488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.629061 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.653815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.653650 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.668607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.661338 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcpf\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.668607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.666951 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.668607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.667585 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.684433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.669369 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:25.684433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.683261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.706884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:25.706842 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:26.019482 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:26.019402 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f3270d_a944_4c64_be89_1bc17fcb7a21.slice/crio-1f0af2ee73ca0d12f3dafd2bd082e7bcb10c9f784d6fbf999663676407eda5c8 WatchSource:0}: Error finding container 1f0af2ee73ca0d12f3dafd2bd082e7bcb10c9f784d6fbf999663676407eda5c8: Status 404 returned error can't find the container with id 1f0af2ee73ca0d12f3dafd2bd082e7bcb10c9f784d6fbf999663676407eda5c8 Oct 31 03:15:26.019784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.019756 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:15:26.023678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.023634 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:26.023817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.023739 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:26.023817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.023790 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:26.079724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.079681 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:26.728732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.728692 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerStarted","Data":"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691"} Oct 31 03:15:26.729139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.728756 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerStarted","Data":"1f0af2ee73ca0d12f3dafd2bd082e7bcb10c9f784d6fbf999663676407eda5c8"} Oct 31 03:15:26.752385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.752332 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podStartSLOduration=1.752316762 podStartE2EDuration="1.752316762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:26.749929747 +0000 UTC m=+1204.604946887" watchObservedRunningTime="2026-10-31 03:15:26.752316762 +0000 UTC m=+1204.607333903" Oct 31 03:15:26.925439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:26.925219 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:27.006141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.006006 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:15:27.708143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.708096 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:27.715172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.715144 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:27.804438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.804160 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:27.814442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.810659 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:15:27.963623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:27.963470 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.111716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:29.111611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.389498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:29.384399 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.533349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:29.533301 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.695437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:29.695013 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.057487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:30.054518 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.601247 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:30.601212 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.792959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:30.788060 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.221432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:31.221390 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.804897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:31.804838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.937283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:31.937247 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:32.461639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:32.461606 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:33.227130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:33.227072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.010814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:34.010776 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.160485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:34.153038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.726050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:34.725937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.031709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.031605 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck"] Oct 31 03:15:35.036760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.036727 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5763-f8sck"] Oct 31 03:15:35.411820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.411783 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.540916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.540878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.837343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.836581 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:15:35.839737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.839686 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:35.839970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.839936 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:35.840181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.840166 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:35.841179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.841161 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:35.841336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.841322 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:35.841472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.841458 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c274a10-83bb-406a-a3b8-4a358563d5f0" containerName="cleanup" Oct 31 03:15:35.841908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.841876 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ab58e5-f40f-4d51-b585-cde97b0799e2" containerName="cleanup" Oct 31 03:15:35.842071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.842041 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8df3b11-da38-4e18-8218-17555a883aaf" containerName="cleanup" Oct 31 03:15:35.842223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.842195 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab481662-5ab8-4a37-979f-2692e1595ab3" containerName="cleanup" Oct 31 03:15:35.847435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.847396 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:35.852540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.852476 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2ad1f0-openshift-default-dockercfg-9k6fq\"" Oct 31 03:15:35.855171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.855149 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:15:35.888383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:35.888344 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:36.005789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.005747 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.006066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.006041 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.006194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.006180 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.070642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.070598 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmq8\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.070838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.070657 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.070956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.070932 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.071022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.070974 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.071081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.071034 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.078441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.071138 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.199756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.199677 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.200018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.199995 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.200218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.200185 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.200408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.200385 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.200673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.200645 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.201049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.201023 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.201252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.201229 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.201400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.201384 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.201668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.201649 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmq8\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.201796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.201780 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.202584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.202562 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.203003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.202979 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.211295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.207438 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.211295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.207751 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.220762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.213960 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.220762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.214733 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.274237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.259102 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmq8\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.275495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.275403 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\") pod \"e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:36.297663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.297619 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:36.306747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.306709 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bba92f-ca29-4985-be28-8d1a6fca7ae1" path="/var/lib/kubelet/pods/18bba92f-ca29-4985-be28-8d1a6fca7ae1/volumes" Oct 31 03:15:36.451311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.451228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:36.492926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:36.492881 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:37.131377 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:37.131332 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ca4c51_2abf_4bf1_9706_08d321b33d5a.slice/crio-64e53b706d99d2ce42c208c7ad65a957667864e97a79dc15dd9ebd699327c14c WatchSource:0}: Error finding container 64e53b706d99d2ce42c208c7ad65a957667864e97a79dc15dd9ebd699327c14c: Status 404 returned error can't find the container with id 64e53b706d99d2ce42c208c7ad65a957667864e97a79dc15dd9ebd699327c14c Oct 31 03:15:37.132337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.132310 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:15:37.134058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.134026 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:37.134161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.134093 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:37.134161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.134124 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:37.210759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.210728 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:37.534575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.534482 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerStarted","Data":"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268"} Oct 31 03:15:37.534735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.534594 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerStarted","Data":"64e53b706d99d2ce42c208c7ad65a957667864e97a79dc15dd9ebd699327c14c"} Oct 31 03:15:37.558762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.558694 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podStartSLOduration=2.558678448 podStartE2EDuration="2.558678448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:37.555072376 +0000 UTC m=+1215.410089517" watchObservedRunningTime="2026-10-31 03:15:37.558678448 +0000 UTC m=+1215.413695588" Oct 31 03:15:38.005207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:37.997409 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:38.494495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:38.494390 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:38.500204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:38.500177 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:38.544207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:38.544176 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:38.545014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:38.544992 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:15:38.705512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:38.705474 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:39.029680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:39.029646 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l"] Oct 31 03:15:39.035623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:39.035594 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009d534b-nnl7l"] Oct 31 03:15:39.040934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:39.040915 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc"] Oct 31 03:15:39.045617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:39.045592 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ef0e5f-2xlhc"] Oct 31 03:15:39.654453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:39.653768 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:40.142312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.142214 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:40.262622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.262585 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558c4870-35a0-454d-bd3f-cb598fc258f6" path="/var/lib/kubelet/pods/558c4870-35a0-454d-bd3f-cb598fc258f6/volumes" Oct 31 03:15:40.263217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.263193 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e839526f-1ab3-481f-9d85-22f4a3ae9f01" path="/var/lib/kubelet/pods/e839526f-1ab3-481f-9d85-22f4a3ae9f01/volumes" Oct 31 03:15:40.565579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.565539 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:15:40.565755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.565606 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:15:40.710838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.710805 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:40.851898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.851798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:40.995974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:40.995937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:41.504778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.504740 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:41.672897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.672832 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:41.923188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.923157 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:41.933663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.933620 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:15:41.961465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.958649 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:15:41.961465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.958762 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:41.964682 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:41.964651 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ad1f0-dockercfg-54dz7\"" Oct 31 03:15:42.039056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.038999 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ad1f0\"" Oct 31 03:15:42.061112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.061072 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg"] Oct 31 03:15:42.073031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.072967 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3977ac75-xfwdg"] Oct 31 03:15:42.102541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.101907 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:42.125923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.125454 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqv8\" (UniqueName: \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.125923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.125759 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.237496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.232101 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.237496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.232381 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqv8\" (UniqueName: \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.237496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.233739 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.277090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.263943 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:42.277090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.269262 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqv8\" (UniqueName: \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\") pod \"payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.280839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.280804 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d4474a-67ca-45f3-b289-7ed33204d44d" path="/var/lib/kubelet/pods/14d4474a-67ca-45f3-b289-7ed33204d44d/volumes" Oct 31 03:15:42.339980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.335289 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:42.604991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.604952 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:42.882397 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:42.882364 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb4c8aa_7cd9_4d25_a887_705f34402642.slice/crio-ec4a99ba13ec0a017060e55791d52321fba420f34d0decbb9e4635327b290a8f WatchSource:0}: Error finding container ec4a99ba13ec0a017060e55791d52321fba420f34d0decbb9e4635327b290a8f: Status 404 returned error can't find the container with id ec4a99ba13ec0a017060e55791d52321fba420f34d0decbb9e4635327b290a8f Oct 31 03:15:42.884064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:42.884032 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:15:43.222397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:43.220851 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:43.614724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:43.614692 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" event={"ID":"cbb4c8aa-7cd9-4d25-a887-705f34402642","Type":"ContainerStarted","Data":"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87"} Oct 31 03:15:43.614870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:43.614751 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" event={"ID":"cbb4c8aa-7cd9-4d25-a887-705f34402642","Type":"ContainerStarted","Data":"ec4a99ba13ec0a017060e55791d52321fba420f34d0decbb9e4635327b290a8f"} Oct 31 03:15:43.947943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:43.946583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:44.623108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:44.623064 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:44.644172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:44.644114 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" podStartSLOduration=3.644092391 podStartE2EDuration="3.644092391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:44.639949194 +0000 UTC m=+1222.494966336" watchObservedRunningTime="2026-10-31 03:15:44.644092391 +0000 UTC m=+1222.499109533" Oct 31 03:15:44.646381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:44.646352 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:45.124289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.124251 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj"] Oct 31 03:15:45.259211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.259172 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj"] Oct 31 03:15:45.259459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.259350 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.262504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.262476 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:15:45.262627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.262553 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:15:45.306335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.306300 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-765ps\" (UniqueName: \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.306697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.306675 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.306844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.306715 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.408322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.408283 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-765ps\" (UniqueName: \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.408593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.408573 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.408677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.408613 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.409308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.409282 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.411336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.411310 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.420658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.420631 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-765ps\" (UniqueName: \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\") pod \"maas-api-revoke-sub-b42ac26a-qr6lj\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.546782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.546741 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:45.691719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.691629 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:45.983989 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:15:45.983960 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27aacf07_8d5c_42ba_a035_1b7afdaba631.slice/crio-be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a WatchSource:0}: Error finding container be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a: Status 404 returned error can't find the container with id be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a Oct 31 03:15:45.986686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:45.986656 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj"] Oct 31 03:15:46.176902 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:15:46.176772 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:15:46.345491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:46.344601 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:46.648142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:46.648108 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" event={"ID":"27aacf07-8d5c-42ba-a035-1b7afdaba631","Type":"ContainerStarted","Data":"81476fa6fb69124bcc2d0644883f3ec4bb3fee72d888275f52be11b4a858ee74"} Oct 31 03:15:46.648309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:46.648158 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" event={"ID":"27aacf07-8d5c-42ba-a035-1b7afdaba631","Type":"ContainerStarted","Data":"be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a"} Oct 31 03:15:46.674766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:46.674710 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" podStartSLOduration=1.674695901 podStartE2EDuration="1.674695901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:46.667392333 +0000 UTC m=+1224.522409477" watchObservedRunningTime="2026-10-31 03:15:46.674695901 +0000 UTC m=+1224.529713038" Oct 31 03:15:47.009273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.009218 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:47.050810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.049793 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z"] Oct 31 03:15:47.053870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.053842 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116c31a2-gcb6z"] Oct 31 03:15:47.660323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.660290 4556 generic.go:356] "Generic (PLEG): container finished" podID="27aacf07-8d5c-42ba-a035-1b7afdaba631" containerID="81476fa6fb69124bcc2d0644883f3ec4bb3fee72d888275f52be11b4a858ee74" exitCode=0 Oct 31 03:15:47.660540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.660385 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" event={"ID":"27aacf07-8d5c-42ba-a035-1b7afdaba631","Type":"ContainerDied","Data":"81476fa6fb69124bcc2d0644883f3ec4bb3fee72d888275f52be11b4a858ee74"} Oct 31 03:15:47.976407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:47.976331 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:48.262169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.262090 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a" path="/var/lib/kubelet/pods/55d44479-d6ba-4c0c-a2dc-d5d44fa9d20a/volumes" Oct 31 03:15:48.439858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.439822 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:48.939607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.939534 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:48.994599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.994561 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\") pod \"27aacf07-8d5c-42ba-a035-1b7afdaba631\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " Oct 31 03:15:48.995027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.994974 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\") pod \"27aacf07-8d5c-42ba-a035-1b7afdaba631\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " Oct 31 03:15:48.995440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.995407 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\" (UniqueName: \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\") pod \"27aacf07-8d5c-42ba-a035-1b7afdaba631\" (UID: \"27aacf07-8d5c-42ba-a035-1b7afdaba631\") " Oct 31 03:15:48.996160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.996127 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca" pod "27aacf07-8d5c-42ba-a035-1b7afdaba631" (UID: "27aacf07-8d5c-42ba-a035-1b7afdaba631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:15:48.996937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:48.996914 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:49.018716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.018676 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps" pod "27aacf07-8d5c-42ba-a035-1b7afdaba631" (UID: "27aacf07-8d5c-42ba-a035-1b7afdaba631"). InnerVolumeSpecName "kube-api-access-765ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:15:49.024947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.024889 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca" pod "27aacf07-8d5c-42ba-a035-1b7afdaba631" (UID: "27aacf07-8d5c-42ba-a035-1b7afdaba631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:15:49.033528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.033497 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b"] Oct 31 03:15:49.040460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.039800 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7c26c39-trc9b"] Oct 31 03:15:49.099097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.099060 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27aacf07-8d5c-42ba-a035-1b7afdaba631-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:49.099097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.099091 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-765ps\" (UniqueName: \"kubernetes.io/projected/27aacf07-8d5c-42ba-a035-1b7afdaba631-kube-api-access-765ps\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:15:49.144867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.144826 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:49.680713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.680683 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" Oct 31 03:15:49.882890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.680711 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj" event={"ID":"27aacf07-8d5c-42ba-a035-1b7afdaba631","Type":"ContainerDied","Data":"be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a"} Oct 31 03:15:49.882890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.680758 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be15d5aa9372a9357f78814575954a3260bfc71c5b2c45d77278623bde220b9a" Oct 31 03:15:49.882890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:49.822860 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:50.036076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.036041 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x"] Oct 31 03:15:50.042941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.042911 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9515da-hlv2x"] Oct 31 03:15:50.070125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.070091 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:15:50.071303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.071276 4556 cpu_manager.go:414] "containerMap: removing container" podUID="27aacf07-8d5c-42ba-a035-1b7afdaba631" containerName="revoke-keys" Oct 31 03:15:50.071427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.071304 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27aacf07-8d5c-42ba-a035-1b7afdaba631" containerName="revoke-keys" Oct 31 03:15:50.071620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.071603 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="27aacf07-8d5c-42ba-a035-1b7afdaba631" containerName="revoke-keys" Oct 31 03:15:50.077058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.077036 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.093478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.093447 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:15:50.116013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.115979 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116152 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116305 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbxh\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116388 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116559 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116537 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116608 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116674 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116785 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.116947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.116844 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218423 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218388 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218463 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218492 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218539 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218572 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218677 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218784 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218848 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.218872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218865 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbxh\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.219086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218880 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.219086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.218964 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.219086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.219030 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.219256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.219211 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.219659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.219637 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.220840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.220823 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.221160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.221143 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.229676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.229650 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.229842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.229824 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbxh\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.260685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.260611 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295b2968-837c-4e0e-98fd-cdd5b9b44f40" path="/var/lib/kubelet/pods/295b2968-837c-4e0e-98fd-cdd5b9b44f40/volumes" Oct 31 03:15:50.261281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.261259 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b7126c-a616-472a-a4d6-aab3dd3a3944" path="/var/lib/kubelet/pods/f9b7126c-a616-472a-a4d6-aab3dd3a3944/volumes" Oct 31 03:15:50.407027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.406988 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:50.632153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.628405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:50.695130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.695089 4556 generic.go:356] "Generic (PLEG): container finished" podID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerID="3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464" exitCode=0 Oct 31 03:15:50.695678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.695166 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerDied","Data":"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464"} Oct 31 03:15:50.695833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.695813 4556 scope.go:122] "RemoveContainer" containerID="3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464" Oct 31 03:15:50.697871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.697843 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:50.698012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.697907 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:50.698012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.697941 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:50.730589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.730556 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:15:50.734348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.734308 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:50.734502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.734401 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:50.734502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:50.734464 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:15:51.338530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:51.338482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:51.706240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:51.706208 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" event={"ID":"556f9edf-13be-4654-b987-5efdda7df8a5","Type":"ContainerStarted","Data":"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802"} Oct 31 03:15:51.706729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:51.706270 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" event={"ID":"556f9edf-13be-4654-b987-5efdda7df8a5","Type":"ContainerStarted","Data":"cc11e0560f5a34c30e78dfd88ae5f9dd423ff32ed5c8fefd318003f0b2e09ec1"} Oct 31 03:15:51.708627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:51.708601 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerStarted","Data":"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90"} Oct 31 03:15:51.733374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:51.733316 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podStartSLOduration=1.7332985939999999 podStartE2EDuration="1.733298594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:51.731256251 +0000 UTC m=+1229.586273392" watchObservedRunningTime="2026-10-31 03:15:51.733298594 +0000 UTC m=+1229.588315735" Oct 31 03:15:52.126840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.119713 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:15:52.143111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.143063 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:52.143319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.143167 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:52.189438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.166108 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:15:52.263442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.253556 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:52.495007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.494967 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:52.495311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.495280 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:52.611217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.611178 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:52.634599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.634562 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.638128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.637782 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2ad1f0-serving-cert\"" Oct 31 03:15:52.647510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.647489 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:52.647620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.647532 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:15:52.716506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.712996 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.716506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.713108 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.716506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.713208 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.716506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.713342 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrbx\" (UniqueName: \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.735892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.735855 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:52.833497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.832876 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.833497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.833026 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrbx\" (UniqueName: \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.834042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.834009 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.882863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.882837 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.884239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.882127 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrbx\" (UniqueName: \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.884445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.882680 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.884607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.846307 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.888501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.883706 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\") pod \"maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:52.960186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:52.958775 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:53.155563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.154564 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:53.155563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.154646 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:53.419960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.419820 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:53.419960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.419885 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:53.508682 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.508624 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:53.530960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.530910 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:15:53.744732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:53.744644 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" event={"ID":"1fb63b71-f423-4052-9756-4e7258aa6a92","Type":"ContainerStarted","Data":"9e265ac6ea8d15cf3f243a33e10b6ea9ddb1e5162034e19d970471f5b7a12613"} Oct 31 03:15:54.111318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.111268 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:54.117503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.117460 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:54.349435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.347220 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:54.420440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.409347 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:54.420440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.409524 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:54.531440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.525519 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:15:54.755482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.755446 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" event={"ID":"1fb63b71-f423-4052-9756-4e7258aa6a92","Type":"ContainerStarted","Data":"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393"} Oct 31 03:15:54.755878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.755535 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:15:54.826438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:54.820534 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" podStartSLOduration=2.307882615 podStartE2EDuration="2.820510788s" totalImagesPullingTime="512.628173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:54.791071598 +0000 UTC m=+1232.646088760" watchObservedRunningTime="2026-10-31 03:15:54.820510788 +0000 UTC m=+1232.675527930" Oct 31 03:15:55.112493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.101852 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:55.112493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.101914 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:55.372496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.371521 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:55.433133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.432251 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:55.433496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.433458 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:55.630344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.630246 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:55.640816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:55.640779 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:15:56.032874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.032837 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf"] Oct 31 03:15:56.041062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.041023 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64231135-5rpvf"] Oct 31 03:15:56.102890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.102854 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:56.103175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.103145 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:56.423549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.415548 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:56.423549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.415608 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:56.487374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.487333 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ab3b90-619f-413a-a1c0-5f7d1940b296" path="/var/lib/kubelet/pods/d0ab3b90-619f-413a-a1c0-5f7d1940b296/volumes" Oct 31 03:15:56.633390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:56.633346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:57.103289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:57.102811 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:57.103289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:57.102871 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:57.417524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:57.417048 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 31 03:15:57.417524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:57.417184 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 31 03:15:57.531122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:57.531081 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:58.098076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.098036 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 31 03:15:58.098297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.098107 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 31 03:15:58.310110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.310051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:58.414579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.414545 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:58.415082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.415061 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:58.416121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:58.416094 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:15:59.003496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:59.003459 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:59.104827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:59.104793 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:15:59.106154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:59.106127 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:15:59.829198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:59.829165 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:59.986372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:15:59.986340 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:00.338701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:00.338671 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:00.653499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:00.650320 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:00.792155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:00.792106 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:00.799622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:00.799593 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:16:01.459185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:01.457286 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:01.592605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:01.592557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:01.898005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:01.897878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:02.036287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:02.036249 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf"] Oct 31 03:16:02.042370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:02.042333 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71077bf3-n66kf"] Oct 31 03:16:02.182903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:02.178547 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:02.320441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:02.320357 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033af525-76e5-4ab8-af2c-50685297b167" path="/var/lib/kubelet/pods/033af525-76e5-4ab8-af2c-50685297b167/volumes" Oct 31 03:16:02.364785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:02.364723 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:03.122616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:03.122573 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:03.921389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:03.921261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:04.103438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:04.102854 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:04.759535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:04.759405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:05.635838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:05.635803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:05.904818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:05.904789 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:06.032673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:06.032625 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:06.433481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:06.433449 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:06.639936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:06.639899 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:06.766363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:06.766272 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:06.894801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:06.894762 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:07.193180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:07.193146 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:07.515682 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:07.515584 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:07.722441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:07.722343 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:08.132358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:08.132325 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:08.257528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:08.257496 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:08.401316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:08.401282 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:08.847990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:08.847958 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:09.492101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:09.492067 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:09.615679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:09.615641 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:10.171185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:10.171147 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:10.302517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:10.302395 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:10.903115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:10.903076 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:11.032947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:11.032901 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:11.302754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:11.302662 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8"] Oct 31 03:16:11.308492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:11.308463 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5168542-bdtd8"] Oct 31 03:16:11.734557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:11.734521 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:11.874525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:11.874494 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:12.228742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:12.228699 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:12.400378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:12.400285 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:12.476436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:12.442794 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1450a0ac-b5f7-4337-bee5-c50dde62649a" path="/var/lib/kubelet/pods/1450a0ac-b5f7-4337-bee5-c50dde62649a/volumes" Oct 31 03:16:12.821598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:12.821561 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:13.581210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:13.581171 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:13.846688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:13.846598 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:13.846852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:13.846719 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:14.031633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:14.031595 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt"] Oct 31 03:16:14.040576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:14.040542 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224ab914-67fzt"] Oct 31 03:16:14.265614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:14.265578 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2068dd00-6f30-464c-9d62-cd7d38956bcb" path="/var/lib/kubelet/pods/2068dd00-6f30-464c-9d62-cd7d38956bcb/volumes" Oct 31 03:16:14.337673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:14.337622 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:15.093090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:15.093052 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr"] Oct 31 03:16:15.124554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:15.124514 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079550a4-9glkr"] Oct 31 03:16:15.249697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:15.249632 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:16.083133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:16.083091 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:16.154998 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:16.154960 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:16:16.286878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:16.286845 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855e6d0e-49f1-4ed1-9b91-80b61e661c04" path="/var/lib/kubelet/pods/855e6d0e-49f1-4ed1-9b91-80b61e661c04/volumes" Oct 31 03:16:16.437402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:16.437363 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:16.568743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:16.568714 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:16.856985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:16.856883 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:17.633163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.633126 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:17.699784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.699746 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:16:17.764649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.764395 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:17.843698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.843651 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:16:17.960437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.845390 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:17.980436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.961857 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-780abd-dockercfg-6ml9j\"" Oct 31 03:16:17.980436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.962280 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-780abd\"" Oct 31 03:16:17.980436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.963241 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:17.980436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:17.963429 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdstm\" (UniqueName: \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:18.072299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.069010 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdstm\" (UniqueName: \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:18.072299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.069372 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:18.105690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.105645 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.523753 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.606220 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.652594 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdstm\" (UniqueName: \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\") pod \"payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.687076 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.689894 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-0a531d-serving-cert\"" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.733008 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.785380 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.785616 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.785722 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.785779 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjg9f\" (UniqueName: \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:18.848158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.807239 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.887889 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.887984 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.888016 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.888047 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjg9f\" (UniqueName: \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.888765 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.978202 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.978237 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjg9f\" (UniqueName: \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.978357 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:18.981865 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:19.033614 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6"] Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:19.038548 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4007e31-cczz6"] Oct 31 03:16:19.226085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:19.144704 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:19.449190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:19.448787 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:16:19.536118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:19.536085 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:20.262306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:20.262269 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b68351c-1856-4feb-8864-63b291e769f7" path="/var/lib/kubelet/pods/8b68351c-1856-4feb-8864-63b291e769f7/volumes" Oct 31 03:16:20.354238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:20.341674 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:20.764538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:20.764505 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:21.248568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:21.241405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:21.939812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:21.939675 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:22.039875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:22.039829 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g"] Oct 31 03:16:22.046073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:22.046043 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2778e280-9ws5g"] Oct 31 03:16:22.264614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:22.264535 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dca2a48-ca2d-4ada-9468-e2fe881bc5d9" path="/var/lib/kubelet/pods/9dca2a48-ca2d-4ada-9468-e2fe881bc5d9/volumes" Oct 31 03:16:22.633496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:22.633368 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.033888 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw"] Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.039507 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79"] Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.044038 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b7c38df-dhknw"] Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.048143 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a7aecd0-7hn79"] Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.212388 4556 scope.go:122] "RemoveContainer" containerID="fdc6d956b660efe6ca981e9dd8387badf719e40dcc3e911271a9830994c19947" Oct 31 03:16:23.742392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:23.468514 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.222630 4556 scope.go:122] "RemoveContainer" containerID="efcfff935b6f27292f0a37a9b2953c1809d2949a772b70c3ffcde758b92afbf2" Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.237217 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.299878 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0807b4bc-d0c9-4008-9017-c5032e50f789" path="/var/lib/kubelet/pods/0807b4bc-d0c9-4008-9017-c5032e50f789/volumes" Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.300684 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4954a30b-9d8b-4416-9954-e6ede8f88354" path="/var/lib/kubelet/pods/4954a30b-9d8b-4416-9954-e6ede8f88354/volumes" Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.424595 4556 scope.go:122] "RemoveContainer" containerID="a9c1da6809cafa99fc2ea5ed0d6f68c4a4bb7fe952b067dec1f6063d7e6e976e" Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.462337 4556 scope.go:122] "RemoveContainer" containerID="b8b536918eb7ee40390ab8bad4612ece90f3476d70fe81593f909abd9f1bbf03" Oct 31 03:16:24.766543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.573239 4556 scope.go:122] "RemoveContainer" containerID="2abef4409dcd871a121b59ae638ecd6d4539f6dcda0ee6ede5aa3665733b942b" Oct 31 03:16:25.011650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.973098 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:16:25.011650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:24.975048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:25.398798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.398645 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:16:25.407667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.407637 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:16:25.498396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.498360 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.648886 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.648992 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.665834 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.665955 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666024 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666043 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666286 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666377 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666426 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666597 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.672033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.666645 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlpd\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.769679 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.769744 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlpd\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.769839 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.769929 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.770029 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.770171 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.770342 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.770408 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.771389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.770469 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.797689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.772972 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.797689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.773610 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.797689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.780915 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.797689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.781169 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.797689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.787656 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:25.824616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:25.824532 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:26.030997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.030904 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8"] Oct 31 03:16:26.035275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.035240 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:26.035502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.035477 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlpd\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:26.042226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.036031 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:26.042226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.036245 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:26.042226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.036249 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c253abb-24kr8"] Oct 31 03:16:26.235796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.234887 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:26.757723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.757687 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c199759-0aca-46cf-a4b6-38f5c32ec4d1" path="/var/lib/kubelet/pods/6c199759-0aca-46cf-a4b6-38f5c32ec4d1/volumes" Oct 31 03:16:26.855335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:26.855294 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:27.877492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:27.877451 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" event={"ID":"f4367dfd-57bb-4a2b-881c-82651b935470","Type":"ContainerStarted","Data":"d231a80819869e92e684cdfb0a68910be87de90423f72fd3bc354dd90034e9f6"} Oct 31 03:16:27.954437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:27.953052 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:28.474436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.467870 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:28.630490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.618613 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" event={"ID":"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8","Type":"ContainerStarted","Data":"eff0849e660ab9a0361bde5f345e91deffba50db81c63b11e08fed6e136350eb"} Oct 31 03:16:28.740488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.718472 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt"] Oct 31 03:16:28.830581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.746272 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d3fc93-zgdpt"] Oct 31 03:16:28.990437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.971709 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:28.990437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:28.984887 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:29.510872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:29.493522 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:29.954793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:29.954754 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" event={"ID":"f4367dfd-57bb-4a2b-881c-82651b935470","Type":"ContainerStarted","Data":"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90"} Oct 31 03:16:29.955021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:29.954856 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:29.975564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:29.975512 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" podStartSLOduration=12.975486983 podStartE2EDuration="12.975486983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:29.971263784 +0000 UTC m=+1267.826280924" watchObservedRunningTime="2026-10-31 03:16:29.975486983 +0000 UTC m=+1267.830504124" Oct 31 03:16:30.212711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.197940 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:16:30.261002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.260951 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f6ccbb-febc-4cfb-95c5-3c5169aa65f5" path="/var/lib/kubelet/pods/98f6ccbb-febc-4cfb-95c5-3c5169aa65f5/volumes" Oct 31 03:16:30.346856 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:16:30.305027 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd134a4_2bb3_479a_b138_75e074a2a039.slice/crio-2b2c8ba118aed02d3ca05a08e527542da95ab7eff1352c89d4d3cd80f5d65a0a WatchSource:0}: Error finding container 2b2c8ba118aed02d3ca05a08e527542da95ab7eff1352c89d4d3cd80f5d65a0a: Status 404 returned error can't find the container with id 2b2c8ba118aed02d3ca05a08e527542da95ab7eff1352c89d4d3cd80f5d65a0a Oct 31 03:16:30.359743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.359671 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:30.359912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.359817 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:30.359912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.359863 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:30.834038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:30.833901 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:31.037300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:31.037257 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" event={"ID":"2bd134a4-2bb3-479a-b138-75e074a2a039","Type":"ContainerStarted","Data":"2b2c8ba118aed02d3ca05a08e527542da95ab7eff1352c89d4d3cd80f5d65a0a"} Oct 31 03:16:31.787585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:31.787544 4556 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6zbtw container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:31.788075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:31.787603 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" podUID="c1d8733a-ce73-455e-aeb6-bba45d34b97f" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:31.885861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:31.885826 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:16:31.987262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:31.987215 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:32.201195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.201150 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:32.323979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.318488 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:32.324266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.324221 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:32.653071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.653035 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:16:32.655743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.655699 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.677477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.674196 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podStartSLOduration=7.674177047 podStartE2EDuration="7.674177047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:32.326027537 +0000 UTC m=+1270.181044682" watchObservedRunningTime="2026-10-31 03:16:32.674177047 +0000 UTC m=+1270.529194189" Oct 31 03:16:32.677477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.675092 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.677477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.675212 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798jb\" (UniqueName: \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.677477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.675409 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.677477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.675541 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.748453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.699690 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-780abd-serving-cert\"" Oct 31 03:16:32.748453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.716950 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" event={"ID":"2bd134a4-2bb3-479a-b138-75e074a2a039","Type":"ContainerStarted","Data":"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d"} Oct 31 03:16:32.748453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.717018 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.777091 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.777143 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798jb\" (UniqueName: \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.777283 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.777319 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:32.777714 4556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-serving-cert: secret "maas-api-e2e-api-b-780abd-serving-cert" not found Oct 31 03:16:32.809864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:32.778343 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:32.868690 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:32.834690 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls podName:630cdffc-badd-42d2-9f09-a0babf994da4 nodeName:}" failed. No retries permitted until 2026-10-31 03:16:33.334664487 +0000 UTC m=+1271.189681609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls") pod "maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4") : secret "maas-api-e2e-api-b-780abd-serving-cert" not found Oct 31 03:16:33.236701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.236666 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:33.236922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.236785 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:33.345147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.345089 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:33.392709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.392672 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:33.392876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.392796 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798jb\" (UniqueName: \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:33.452201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.393334 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:33.452201 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:33.393494 4556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-serving-cert: secret "maas-api-e2e-api-b-780abd-serving-cert" not found Oct 31 03:16:33.452201 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:33.393551 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls podName:630cdffc-badd-42d2-9f09-a0babf994da4 nodeName:}" failed. No retries permitted until 2026-10-31 03:16:34.39353527 +0000 UTC m=+1272.248552391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls") pod "maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4") : secret "maas-api-e2e-api-b-780abd-serving-cert" not found Oct 31 03:16:33.875572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:33.875538 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:34.238575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.221047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:34.242913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.241698 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:34.242913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.241782 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:34.491299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.412842 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:34.491299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.415738 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:34.713554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:34.617332 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:35.114174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.114065 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:35.155561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.155528 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:16:35.243125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.243083 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:35.382611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.243158 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:35.382611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.277432 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" podStartSLOduration=10.785249725 podStartE2EDuration="17.277399923s" totalImagesPullingTime="6.492150198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:35.27302487 +0000 UTC m=+1273.128042015" watchObservedRunningTime="2026-10-31 03:16:35.277399923 +0000 UTC m=+1273.132417065" Oct 31 03:16:35.728774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.728733 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" event={"ID":"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8","Type":"ContainerStarted","Data":"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677"} Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744052 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkpnk\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744111 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744228 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744279 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744314 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744433 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744451 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744769 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:35.797053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.744815 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846198 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846235 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846321 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846337 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846610 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846654 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846680 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846716 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846729 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkpnk\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846768 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846863 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846925 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.846969 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.174570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:35.895999 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.217102 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.218245 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.221239 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkpnk\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.379080 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.379189 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.384195 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.384255 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:36.401527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:36.386223 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:37.517708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.517672 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:37.521392 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.142s" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.525347 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.525373 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.525452 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.527817 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.528067 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.528324 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:16:37.573532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.528601 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:16:37.902143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.735663 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\") pod \"e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:38.062625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.921517 4556 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6zbtw container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:38.062625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.921572 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" podUID="c1d8733a-ce73-455e-aeb6-bba45d34b97f" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:38.062625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:37.972226 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:38.062625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:38.022770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:38.236015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:38.235928 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:38.236015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:38.235987 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:39.027044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.026949 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v"] Oct 31 03:16:39.077180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.031898 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab16c1e0-qrn6v"] Oct 31 03:16:39.205111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.203913 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:39.259736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.235960 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:39.259736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.236030 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:39.948872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:39.948802 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:40.236285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:40.236129 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:40.236285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:40.236222 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:41.106057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.106000 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:41.242236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.242175 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:41.243164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.243134 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:41.252888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.252858 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0c6e7a-d3d2-4e29-8704-049cdf360eb6" path="/var/lib/kubelet/pods/3f0c6e7a-d3d2-4e29-8704-049cdf360eb6/volumes" Oct 31 03:16:41.347990 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:41.347946 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.098s" Oct 31 03:16:41.400800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.348000 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:16:41.588058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.588023 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:16:41.588225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.588164 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.588225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.588184 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:41.588357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.588250 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:16:41.740200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.740107 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.740391 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.740571 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.740907 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.740998 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.741092 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.741163 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.741704 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.767724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.741757 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.829168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.784539 4556 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6zbtw container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:16:41.829168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.784645 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zbtw" podUID="c1d8733a-ce73-455e-aeb6-bba45d34b97f" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.843993 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844078 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844147 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844284 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844334 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844392 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844464 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844544 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844573 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844679 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844863 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844866 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.844897 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.853647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.845051 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:41.989374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:41.988994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:42.234244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.234212 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:42.301274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.234470 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:42.301274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.234525 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:42.301274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.234717 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\") pod \"e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:42.301274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.235994 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:42.301274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.236034 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:42.376776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:42.376734 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:43.016822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.016792 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:16:43.235777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.235741 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:43.236031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.235800 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:43.631493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.516578 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:43.652698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.652611 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:16:43.662881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.662756 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:16:43.693442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.686664 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": read tcp 10.134.0.2:53800->10.134.0.157:15021: read: connection reset by peer" start-of-body= Oct 31 03:16:43.693442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.686747 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": read tcp 10.134.0.2:53800->10.134.0.157:15021: read: connection reset by peer" Oct 31 03:16:43.866510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:43.706011 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:16:44.292628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.235838 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:44.292628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.235897 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:44.406729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.367756 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:16:44.406729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.378229 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.406729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.378331 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.406729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.400541 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.412881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.412838 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.413004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.412930 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.413004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.412972 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.417984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.417958 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" event={"ID":"9cae1473-cb7e-4ed9-ab05-51290cb2e06b","Type":"ContainerStarted","Data":"3bce468a7b7c84a1327225bc27d15cac7558fe0b204a79493589d2f5be57ad64"} Oct 31 03:16:44.423202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.423102 4556 generic.go:356] "Generic (PLEG): container finished" podID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerID="feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691" exitCode=0 Oct 31 03:16:44.423976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.423329 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerDied","Data":"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691"} Oct 31 03:16:44.423976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.423907 4556 scope.go:122] "RemoveContainer" containerID="feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691" Oct 31 03:16:44.434482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.434396 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" event={"ID":"630cdffc-badd-42d2-9f09-a0babf994da4","Type":"ContainerStarted","Data":"65a84958bf0559ff701136843116c4257bd7815757c4b4e72fb516a4fbf1630f"} Oct 31 03:16:44.460464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.459148 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.460464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.459249 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:44.460464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:44.459307 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:16:45.051760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.051724 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:45.091476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.066912 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82"] Oct 31 03:16:45.217473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.209930 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5b94638-tkt82"] Oct 31 03:16:45.485900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.459119 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.485900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.459170 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:45.527076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.527029 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerStarted","Data":"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74"} Oct 31 03:16:45.530522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.530111 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" event={"ID":"c41f8be1-e9fc-478f-96fc-29c9bea88868","Type":"ContainerStarted","Data":"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab"} Oct 31 03:16:45.530670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.530554 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" event={"ID":"c41f8be1-e9fc-478f-96fc-29c9bea88868","Type":"ContainerStarted","Data":"c8445167808f6fc0bd06ff17de77e2c4207cfbe3abb33a10d044bdcceba2053d"} Oct 31 03:16:45.532671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.532640 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" event={"ID":"9cae1473-cb7e-4ed9-ab05-51290cb2e06b","Type":"ContainerStarted","Data":"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399"} Oct 31 03:16:45.660784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.653749 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podStartSLOduration=10.653727825 podStartE2EDuration="10.653727825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:45.647326169 +0000 UTC m=+1283.502343318" watchObservedRunningTime="2026-10-31 03:16:45.653727825 +0000 UTC m=+1283.508744969" Oct 31 03:16:45.708479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.685728 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podStartSLOduration=5.685704921 podStartE2EDuration="5.685704921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:45.677294824 +0000 UTC m=+1283.532311969" watchObservedRunningTime="2026-10-31 03:16:45.685704921 +0000 UTC m=+1283.540722063" Oct 31 03:16:45.714459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.710912 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:16:45.714459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.711038 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.714459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.711084 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:45.985498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.975136 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:45.985498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.976568 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:45.985498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:45.976634 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:46.250493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.249534 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.250493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.249603 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:46.451456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.446268 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.451456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.446334 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:46.460541 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:46.452013 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:16:46.583525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.563126 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7de9b4-3d34-48b1-adf2-df8dd25893cb" path="/var/lib/kubelet/pods/1d7de9b4-3d34-48b1-adf2-df8dd25893cb/volumes" Oct 31 03:16:46.607443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.590445 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:46.736439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.726140 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:46.784989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.784931 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:47.015437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.998790 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.015437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:46.998871 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:47.250439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.249543 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.255443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.250836 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:16:47.285599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.266863 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.285599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.266921 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 03:16:47.285599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.268760 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:16:47.285599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.269138 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.342440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.311829 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" event={"ID":"630cdffc-badd-42d2-9f09-a0babf994da4","Type":"ContainerStarted","Data":"9ca37999fbe287e61363de32586a975280299724d972db121c5c5fe9251954c7"} Oct 31 03:16:47.342440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.334051 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:16:47.342440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.334083 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:16:47.342440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.337430 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:47.351436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.350904 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.366470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.358315 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.410436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.409307 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.410436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.409749 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:16:47.431090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.430637 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:16:47.440896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.440289 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.440896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.440382 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:47.538329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.538206 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" podStartSLOduration=15.496759508 podStartE2EDuration="16.538185285s" totalImagesPullingTime="1.041425777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:47.359764173 +0000 UTC m=+1285.214781317" watchObservedRunningTime="2026-10-31 03:16:47.538185285 +0000 UTC m=+1285.393202428" Oct 31 03:16:47.596450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.583295 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:47.651847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.632557 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:47.689434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.683928 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:16:47.735440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.730556 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:47.735440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.730624 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:47.971489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.928240 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:47.971489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:47.928312 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.038822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.018704 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.133437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.105294 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.155439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.137494 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:48.187606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.167544 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.267843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.266575 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.273803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.273762 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:16:48.304815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.275192 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:16:48.304815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.275293 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:16:48.304815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.293887 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.304815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.299514 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:16:48.337440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.336891 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.368439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.350882 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.368439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.360673 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.431551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.431501 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.457902 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.457856 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:16:48.486484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.441077 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.486484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.485628 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:48.572568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.510487 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.626441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.580714 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:48.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.726832 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:16:48.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.747930 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:48.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.747989 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:48.883495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:48.795735 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.022099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.022053 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.060443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.047454 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.099443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.098246 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:49.312610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.312562 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.328564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.328519 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:16:49.328875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.328850 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.351546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.351501 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:16:49.353112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.353077 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 03:16:49.411997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.395517 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.411997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.395585 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:49.413771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.413730 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.415697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.415659 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 31 03:16:49.420532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.420501 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:16:49.518753 4556 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.138s" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.518946 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.518980 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw"] Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519030 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519066 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519086 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d86f9b-xhmmw"] Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519141 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519173 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519207 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519241 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519273 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519305 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519338 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519371 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:16:49.550214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.519403 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:16:49.719439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.718434 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:16:49.719439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.718862 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:49.719439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.718934 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:49.744436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.720997 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:16:49.744436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.721222 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:16:49.744436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.721432 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:16:49.744436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.721600 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:16:49.871612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.769807 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.871612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.827928 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:49.991632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:49.991514 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.126067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.126027 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:50.126376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.126346 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:50.259586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.259489 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:16:50.325530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.325485 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.359993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.359955 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.443281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.442501 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.451926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.443578 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 31 03:16:50.538670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.538566 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.548248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.548216 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:50.555581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.555545 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:50.697213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.697175 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:16:50.821813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.811685 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 31 03:16:50.821813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.812067 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 03:16:50.853593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.853549 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:50.946538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.930945 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:50.946538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.931006 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:50.956385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.956352 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b830dbac-f3f6-43e3-9293-194fadc7dfd3" path="/var/lib/kubelet/pods/b830dbac-f3f6-43e3-9293-194fadc7dfd3/volumes" Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.974927 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8"] Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.975013 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.975065 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.975111 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.975144 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:16:50.983437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.975177 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" Oct 31 03:16:51.013668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.993108 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:16:51.013668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.993310 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.013668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.995641 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:51.013668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:50.995693 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:51.076490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.075949 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:51.262437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.261809 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.407010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.365102 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0f201f-dksh8"] Oct 31 03:16:51.407010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.366273 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 31 03:16:51.460086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.456698 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:51.460086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.456771 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:51.463343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.463311 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:16:51.608332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.591055 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:51.640943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.639329 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:16:51.866228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.866135 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:16:51.946678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.941808 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:16:51.949467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.878468 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:51.949781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.949755 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:51.997954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:51.997258 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:52.057447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.031966 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:16:52.286477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.270199 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:52.286477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.270279 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:52.578492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.575447 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Oct 31 03:16:52.658203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.649357 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:52.737396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.737343 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:16:52.747706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.747642 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 31 03:16:52.749619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.747727 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 31 03:16:52.760155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.751737 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:52.760155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.751894 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 31 03:16:52.760155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.751958 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 31 03:16:52.760155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.757266 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ead49da-7d35-410e-889f-c17ea8187468" path="/var/lib/kubelet/pods/6ead49da-7d35-410e-889f-c17ea8187468/volumes" Oct 31 03:16:52.765751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.765712 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 31 03:16:52.768382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.767686 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:16:52.768382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.767745 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:16:52.789720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.789688 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:16:52.794399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.794375 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:16:52.980973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.972817 4556 patch_prober.go:29] interesting pod/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 03:16:52.980973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.972877 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 03:16:52.980973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:52.973268 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:53.391361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.391278 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:53.392637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.392611 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:53.393282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.393264 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:16:53.467289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.465643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:53.712004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.711973 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:16:53.712544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.712526 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:16:53.713212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:53.713196 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:16:54.007542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:54.007453 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:54.009652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:54.009296 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:16:54.340851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:54.340727 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:55.137998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:55.137956 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:56.028142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:56.028108 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t"] Oct 31 03:16:56.038877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:56.038832 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c19ba506-4896t"] Oct 31 03:16:56.095061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:56.094997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:56.268387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:56.268349 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594753ca-54dc-42de-9a03-14185f7e32b8" path="/var/lib/kubelet/pods/594753ca-54dc-42de-9a03-14185f7e32b8/volumes" Oct 31 03:16:57.124702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:57.124659 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:57.886105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:57.886051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:58.132409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:58.132374 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:58.915932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:58.915901 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:59.384311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:59.384224 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:16:59.384502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:59.384459 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" containerID="cri-o://1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345" gracePeriod=30 Oct 31 03:16:59.390370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:59.390336 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:16:59.390620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:59.390595 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" containerID="cri-o://a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6" gracePeriod=30 Oct 31 03:16:59.886832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:16:59.886228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:01.009714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:01.008660 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:01.900866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:01.900824 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:02.535429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:02.535362 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:02.661021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:02.660973 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:03.029919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:03.029866 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch"] Oct 31 03:17:03.034917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:03.034884 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce35cb0-dskch"] Oct 31 03:17:03.539251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:03.539145 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:04.233177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.233074 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:04.317751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.317709 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc17f3b-9f7f-4421-a30b-79560e81a844" path="/var/lib/kubelet/pods/8fc17f3b-9f7f-4421-a30b-79560e81a844/volumes" Oct 31 03:17:04.403553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.403516 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:04.535049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.535013 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:04.750927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.750846 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:04.788915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.788880 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:17:04.857572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.857539 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:17:04.889203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889165 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.889397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889219 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.889397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889258 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.889397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889299 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.889397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889348 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889446 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889490 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889555 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889614 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889721 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889822 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889849 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.889972 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.890067 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.890102 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.890129 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.890186 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\") pod \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\" (UID: \"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab\") " Oct 31 03:17:04.890367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.890271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\") pod \"56271477-1bf7-48ae-a53f-bcd2f33afe18\" (UID: \"56271477-1bf7-48ae-a53f-bcd2f33afe18\") " Oct 31 03:17:04.894024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.893989 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.899337 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.899378 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.900183 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.900238 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.901253 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.901743 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.902704 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.902733 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.903495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.903430 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.910251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.909679 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.910251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.909786 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "kube-api-access-tb5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:04.910912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.910889 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:04.911924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.911711 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:04.912577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.912560 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:04.913283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.913265 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "kube-api-access-qrrt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:04.913684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.913663 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token" pod "56271477-1bf7-48ae-a53f-bcd2f33afe18" (UID: "56271477-1bf7-48ae-a53f-bcd2f33afe18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:04.913743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:04.913703 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token" pod "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" (UID: "d0dbc58d-dba2-4c13-92d5-00b08f9c78ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:05.004835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.004726 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.004835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.004771 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.006879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006840 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.006879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006872 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006887 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006904 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006916 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006929 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006944 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrt8\" (UniqueName: \"kubernetes.io/projected/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-kube-api-access-qrrt8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006960 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006974 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56271477-1bf7-48ae-a53f-bcd2f33afe18-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006987 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56271477-1bf7-48ae-a53f-bcd2f33afe18-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.006998 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.007007 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb5qp\" (UniqueName: \"kubernetes.io/projected/56271477-1bf7-48ae-a53f-bcd2f33afe18-kube-api-access-tb5qp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.007017 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56271477-1bf7-48ae-a53f-bcd2f33afe18-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.007028 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.007044 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.008223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.007114 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:05.030827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.030780 4556 generic.go:356] "Generic (PLEG): container finished" podID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerID="a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6" exitCode=0 Oct 31 03:17:05.031038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.030860 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" Oct 31 03:17:05.031038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.030851 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" event={"ID":"56271477-1bf7-48ae-a53f-bcd2f33afe18","Type":"ContainerDied","Data":"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6"} Oct 31 03:17:05.031038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.031010 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw" event={"ID":"56271477-1bf7-48ae-a53f-bcd2f33afe18","Type":"ContainerDied","Data":"89e70e92357e935b796bb22bafa010483fce14975aec98e797cfba5ec3d02744"} Oct 31 03:17:05.031227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.031051 4556 scope.go:122] "RemoveContainer" containerID="a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6" Oct 31 03:17:05.035143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.035111 4556 generic.go:356] "Generic (PLEG): container finished" podID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerID="1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345" exitCode=0 Oct 31 03:17:05.035260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.035177 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" Oct 31 03:17:05.035260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.035172 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" event={"ID":"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab","Type":"ContainerDied","Data":"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345"} Oct 31 03:17:05.035260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.035245 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz" event={"ID":"d0dbc58d-dba2-4c13-92d5-00b08f9c78ab","Type":"ContainerDied","Data":"23e8a96c60b443b8cd4c1e91f8339982335123978b4b80faafcd7026ef58e685"} Oct 31 03:17:05.058944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.058919 4556 scope.go:122] "RemoveContainer" containerID="a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6" Oct 31 03:17:05.060874 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:05.060847 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6\": container with ID starting with a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6 not found: ID does not exist" containerID="a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6" Oct 31 03:17:05.060955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.060885 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6"} err="failed to get container status \"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6\": rpc error: code = NotFound desc = could not find container \"a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6\": container with ID starting with a1ee941b412c8c2e20e13be901b7eadf8e3295f5c99cb66b099417e271a340a6 not found: ID does not exist" Oct 31 03:17:05.060955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.060905 4556 scope.go:122] "RemoveContainer" containerID="1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345" Oct 31 03:17:05.068780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.068729 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:17:05.073912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.073882 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-822xw"] Oct 31 03:17:05.078076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.078055 4556 scope.go:122] "RemoveContainer" containerID="1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345" Oct 31 03:17:05.078670 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:05.078404 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345\": container with ID starting with 1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345 not found: ID does not exist" containerID="1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345" Oct 31 03:17:05.078670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.078448 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345"} err="failed to get container status \"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345\": rpc error: code = NotFound desc = could not find container \"1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345\": container with ID starting with 1961ca0a7987cb9cb71938387faf97ca28395145aaa03f4a197a906e341b5345 not found: ID does not exist" Oct 31 03:17:05.082896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.082867 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:17:05.087818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.087789 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-bxpqz"] Oct 31 03:17:05.145813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.145772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:05.181473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.181429 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:17:05.182802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.182776 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" Oct 31 03:17:05.182889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.182806 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" Oct 31 03:17:05.182889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.182868 4556 cpu_manager.go:414] "containerMap: removing container" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" Oct 31 03:17:05.182889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.182878 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" Oct 31 03:17:05.183114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.183098 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" containerName="istio-proxy" Oct 31 03:17:05.183199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.183135 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" containerName="istio-proxy" Oct 31 03:17:05.187971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.187942 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.218491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.218455 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:17:05.453456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453016 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.453456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453109 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.453456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453293 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.453456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453378 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.453456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453449 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.454494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453522 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.454494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453651 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t462f\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.454494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453795 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.454494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.453896 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.540461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.540402 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573112 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573193 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573250 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573374 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t462f\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573522 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573588 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573638 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573711 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.573910 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.602143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.575368 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.686925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.681745 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b93a15-openshift-default-dockercfg-6btpn\"" Oct 31 03:17:05.699161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.699077 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.699968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.699919 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.700202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.700050 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.700993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.700935 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.703094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.703007 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.706990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.706884 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t462f\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.708855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.708800 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.709083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.709041 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\") pod \"e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.805961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.805913 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:05.925839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:05.925766 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:06.113887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.113857 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:17:06.118023 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:06.117991 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738565e2_9595_4697_afee_2838686a2645.slice/crio-ebe9e16c624936553818b8800d216f6dded12e4eabdc2db9a93bf5c1733e0c92 WatchSource:0}: Error finding container ebe9e16c624936553818b8800d216f6dded12e4eabdc2db9a93bf5c1733e0c92: Status 404 returned error can't find the container with id ebe9e16c624936553818b8800d216f6dded12e4eabdc2db9a93bf5c1733e0c92 Oct 31 03:17:06.121558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.121523 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:06.121666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.121608 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:06.121666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.121641 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:06.235524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.233160 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:06.293157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.293117 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56271477-1bf7-48ae-a53f-bcd2f33afe18" path="/var/lib/kubelet/pods/56271477-1bf7-48ae-a53f-bcd2f33afe18/volumes" Oct 31 03:17:06.293952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:06.293931 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0dbc58d-dba2-4c13-92d5-00b08f9c78ab" path="/var/lib/kubelet/pods/d0dbc58d-dba2-4c13-92d5-00b08f9c78ab/volumes" Oct 31 03:17:07.064215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.064175 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerStarted","Data":"fa02f068b53acac7a319cde8ee1ac4308fbb039bf5d53a82bee64cd596639637"} Oct 31 03:17:07.064722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.064246 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerStarted","Data":"ebe9e16c624936553818b8800d216f6dded12e4eabdc2db9a93bf5c1733e0c92"} Oct 31 03:17:07.085961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.085894 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podStartSLOduration=2.085874329 podStartE2EDuration="2.085874329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:07.085540532 +0000 UTC m=+1304.940557672" watchObservedRunningTime="2026-10-31 03:17:07.085874329 +0000 UTC m=+1304.940891471" Oct 31 03:17:07.295424 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.295376 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:07.817438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.811566 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:07.835702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:07.835653 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:08.030634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.030596 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p"] Oct 31 03:17:08.040434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.038965 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c62210-pw29p"] Oct 31 03:17:08.074201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.074110 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:08.076222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.076198 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:17:08.546726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.546663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:08.610757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:08.610710 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014ee252-8356-4f00-9c7e-c10250efa491" path="/var/lib/kubelet/pods/014ee252-8356-4f00-9c7e-c10250efa491/volumes" Oct 31 03:17:09.100369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:09.100316 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:09.699092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:09.699051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:09.887093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:09.887049 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:11.080441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:11.032029 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:17:11.166316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:11.166277 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:11.352887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:11.352798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:12.682452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:12.680020 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:13.011096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:13.010955 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:13.142472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:13.142392 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:13.916459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:13.916399 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:14.128656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:14.128601 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:17:14.128930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:14.128897 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" containerID="cri-o://d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff" gracePeriod=30 Oct 31 03:17:14.136923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:14.136891 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:17:14.137812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:14.137143 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" containerID="cri-o://165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997" gracePeriod=30 Oct 31 03:17:14.940083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:14.940048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:15.394219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:15.394121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:15.794006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:15.793966 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:15.929764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:15.929725 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:15.948961 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:15.948919 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:17:16.097848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.097704 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" podUID="cbb4c8aa-7cd9-4d25-a887-705f34402642" containerName="payload-processing" containerID="cri-o://465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87" gracePeriod=25 Oct 31 03:17:16.459538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.459504 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:17:16.553918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.553881 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\") pod \"cbb4c8aa-7cd9-4d25-a887-705f34402642\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " Oct 31 03:17:16.554294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.554271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\" (UniqueName: \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\") pod \"cbb4c8aa-7cd9-4d25-a887-705f34402642\" (UID: \"cbb4c8aa-7cd9-4d25-a887-705f34402642\") " Oct 31 03:17:16.554981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.554955 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume" pod "cbb4c8aa-7cd9-4d25-a887-705f34402642" (UID: "cbb4c8aa-7cd9-4d25-a887-705f34402642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:16.555374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.555351 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb4c8aa-7cd9-4d25-a887-705f34402642-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:16.557694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.557669 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8" pod "cbb4c8aa-7cd9-4d25-a887-705f34402642" (UID: "cbb4c8aa-7cd9-4d25-a887-705f34402642"). InnerVolumeSpecName "kube-api-access-8fqv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:16.624490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.624463 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:16.659464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.659421 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fqv8\" (UniqueName: \"kubernetes.io/projected/cbb4c8aa-7cd9-4d25-a887-705f34402642-kube-api-access-8fqv8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:16.715372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.715262 4556 generic.go:356] "Generic (PLEG): container finished" podID="cbb4c8aa-7cd9-4d25-a887-705f34402642" containerID="465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87" exitCode=0 Oct 31 03:17:16.715622 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.715606 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" Oct 31 03:17:16.715696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.715641 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" event={"ID":"cbb4c8aa-7cd9-4d25-a887-705f34402642","Type":"ContainerDied","Data":"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87"} Oct 31 03:17:16.715756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.715709 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl" event={"ID":"cbb4c8aa-7cd9-4d25-a887-705f34402642","Type":"ContainerDied","Data":"ec4a99ba13ec0a017060e55791d52321fba420f34d0decbb9e4635327b290a8f"} Oct 31 03:17:16.715756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.715733 4556 scope.go:122] "RemoveContainer" containerID="465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87" Oct 31 03:17:16.741640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.735581 4556 scope.go:122] "RemoveContainer" containerID="465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87" Oct 31 03:17:16.741640 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:16.736069 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87\": container with ID starting with 465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87 not found: ID does not exist" containerID="465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87" Oct 31 03:17:16.741640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.736106 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87"} err="failed to get container status \"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87\": rpc error: code = NotFound desc = could not find container \"465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87\": container with ID starting with 465432c042c31f4aeb7da0974e72afae81ea335cb1b0a70cf3043df221960c87 not found: ID does not exist" Oct 31 03:17:16.760999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.760041 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:17:16.783439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:16.768100 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad1f0-6bd647f7ff-gb6dl"] Oct 31 03:17:17.767795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:17.761260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:18.027847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:18.027761 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml"] Oct 31 03:17:18.032463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:18.032388 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-205cd79e-bzlml"] Oct 31 03:17:18.263857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:18.263818 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e269f2-8bf4-41c2-bcb3-426fdee10a7f" path="/var/lib/kubelet/pods/30e269f2-8bf4-41c2-bcb3-426fdee10a7f/volumes" Oct 31 03:17:18.266372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:18.266344 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb4c8aa-7cd9-4d25-a887-705f34402642" path="/var/lib/kubelet/pods/cbb4c8aa-7cd9-4d25-a887-705f34402642/volumes" Oct 31 03:17:18.623464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:18.623399 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:19.153336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.153302 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": read tcp 10.134.0.2:38524->10.134.0.121:15021: read: connection reset by peer" start-of-body= Oct 31 03:17:19.154660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.153881 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": read tcp 10.134.0.2:38524->10.134.0.121:15021: read: connection reset by peer" Oct 31 03:17:19.520381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.510709 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:19.575567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.565628 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:17:19.633989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.633955 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.677781 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.677839 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.677922 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.677973 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678014 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678044 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678078 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678110 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678146 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678256 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678307 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.678555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678526 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678566 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678661 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678768 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\") pod \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\" (UID: \"8fcd2182-c5c4-417d-b08b-4afee41ed04f\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678881 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.678968 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.679031 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\") pod \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\" (UID: \"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323\") " Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.681487 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.682359 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.682528 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.682693 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.683193 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.683468 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.683695 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.683815 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.684303 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.686754 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.690810 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.690908 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.691648 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.691910 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "kube-api-access-kp6ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.692180 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.692695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.692327 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "kube-api-access-cfdl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:19.694775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.693342 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo" pod "8fcd2182-c5c4-417d-b08b-4afee41ed04f" (UID: "8fcd2182-c5c4-417d-b08b-4afee41ed04f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:19.694775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.694196 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy" pod "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" (UID: "fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:19.768605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.768557 4556 generic.go:356] "Generic (PLEG): container finished" podID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerID="165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997" exitCode=0 Oct 31 03:17:19.768837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.768742 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" Oct 31 03:17:19.768837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.768828 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" event={"ID":"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323","Type":"ContainerDied","Data":"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997"} Oct 31 03:17:19.768961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.768923 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s" event={"ID":"fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323","Type":"ContainerDied","Data":"2f222f9c8cc94dd717afa435759e597f638e2ce08522dbe59d7a4d34487aa573"} Oct 31 03:17:19.769011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.768962 4556 scope.go:122] "RemoveContainer" containerID="165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997" Oct 31 03:17:19.792021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.791983 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6ks\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-kube-api-access-kp6ks\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792019 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792031 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792046 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdl8\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-kube-api-access-cfdl8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792057 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792066 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792083 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792096 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792105 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792114 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792128 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792137 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792146 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fcd2182-c5c4-417d-b08b-4afee41ed04f-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792157 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792175 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792184 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792193 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.792288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.792202 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fcd2182-c5c4-417d-b08b-4afee41ed04f-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:19.795592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.795559 4556 generic.go:356] "Generic (PLEG): container finished" podID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerID="d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff" exitCode=0 Oct 31 03:17:19.795757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.795655 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" Oct 31 03:17:19.795896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.795804 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" event={"ID":"8fcd2182-c5c4-417d-b08b-4afee41ed04f","Type":"ContainerDied","Data":"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff"} Oct 31 03:17:19.795896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.795852 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc" event={"ID":"8fcd2182-c5c4-417d-b08b-4afee41ed04f","Type":"ContainerDied","Data":"5b17666ab27ec5a1c733047d76e58e35ac2726e5de3dc15bcf9f572a4011792d"} Oct 31 03:17:19.811778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.811748 4556 scope.go:122] "RemoveContainer" containerID="165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997" Oct 31 03:17:19.812152 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:19.812108 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997\": container with ID starting with 165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997 not found: ID does not exist" containerID="165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997" Oct 31 03:17:19.812254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.812161 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997"} err="failed to get container status \"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997\": rpc error: code = NotFound desc = could not find container \"165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997\": container with ID starting with 165f4da213c9eaa8d9db32580d443caa4e89ca913ac0f3870669852563bee997 not found: ID does not exist" Oct 31 03:17:19.812254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.812187 4556 scope.go:122] "RemoveContainer" containerID="d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff" Oct 31 03:17:19.820360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.820330 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:17:19.824517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.824483 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-cqz7s"] Oct 31 03:17:19.836646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.836620 4556 scope.go:122] "RemoveContainer" containerID="d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff" Oct 31 03:17:19.837081 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:19.837049 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff\": container with ID starting with d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff not found: ID does not exist" containerID="d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff" Oct 31 03:17:19.837266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.837092 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff"} err="failed to get container status \"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff\": rpc error: code = NotFound desc = could not find container \"d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff\": container with ID starting with d09a51b3b5a3e36a63edeff665cf80feac6e202d8fbef756c3e249081d35d0ff not found: ID does not exist" Oct 31 03:17:19.841360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.841280 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:17:19.846147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:19.846113 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-zkpgc"] Oct 31 03:17:20.262659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:20.262626 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" path="/var/lib/kubelet/pods/8fcd2182-c5c4-417d-b08b-4afee41ed04f/volumes" Oct 31 03:17:20.264380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:20.264356 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" path="/var/lib/kubelet/pods/fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323/volumes" Oct 31 03:17:20.336388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:20.332139 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:21.138730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:21.138688 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:21.907037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:21.907005 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:22.726141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:22.726095 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:23.563737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:23.563693 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:24.355435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:24.347721 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:24.612667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:24.612577 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:17:24.612667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:24.612652 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:17:24.645859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:24.645811 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:25.035442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.027367 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:17:25.035442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.027658 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" podUID="196598da-e0d8-4686-8ba0-069617cd19ad" containerName="maas-api" containerID="cri-o://aa855de6d01c7edf4fd06a599a8416b52034c6b69bfa419eb41f951bc2172d44" gracePeriod=30 Oct 31 03:17:25.126338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.104605 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:25.289663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.289517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:25.569072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.568986 4556 scope.go:122] "RemoveContainer" containerID="dd4f9f13f7aa37d8c11ebfcb41335cc2fc8cb2b664cba78e19a6fdbccc28b5dc" Oct 31 03:17:25.595231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.595201 4556 scope.go:122] "RemoveContainer" containerID="b7e73d6ade03d09446099f3805280ea5a97406b4f67b0dfe509f82420b9bff3c" Oct 31 03:17:25.623401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.623375 4556 scope.go:122] "RemoveContainer" containerID="966f0cd81e72cf668a31242a22995a552f308e27b2abe7ce4501f0f5fd539ae0" Oct 31 03:17:25.649430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.649383 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:25.685500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.683427 4556 scope.go:122] "RemoveContainer" containerID="302b2d11c995f2667f221b366cd43b29f76a2da508824ebd8fad2b11e0850178" Oct 31 03:17:25.714486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.714458 4556 scope.go:122] "RemoveContainer" containerID="0cad4199ec96e430ebc991af7f518bda9b82b9bcc7500b024d324aca47e92610" Oct 31 03:17:25.745768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.745737 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.746070 4556 scope.go:122] "RemoveContainer" containerID="6013b1fd1b96524c24037c33ba19deb5a4b18dc2f1746f8046a4b257fbb05289" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747192 4556 cpu_manager.go:414] "containerMap: removing container" podUID="cbb4c8aa-7cd9-4d25-a887-705f34402642" containerName="payload-processing" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747212 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb4c8aa-7cd9-4d25-a887-705f34402642" containerName="payload-processing" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747228 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747236 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747312 4556 cpu_manager.go:414] "containerMap: removing container" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747320 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747512 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba3bee4-df5b-4a7c-9c0e-b9a5aa40a323" containerName="istio-proxy" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747582 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb4c8aa-7cd9-4d25-a887-705f34402642" containerName="payload-processing" Oct 31 03:17:25.751986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.747596 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcd2182-c5c4-417d-b08b-4afee41ed04f" containerName="istio-proxy" Oct 31 03:17:25.803867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.789943 4556 scope.go:122] "RemoveContainer" containerID="79dc160d45ed293be28c55e884f1ffe41c101ceadb109c215346667f40d22e7f" Oct 31 03:17:25.803867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.802409 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:17:25.803867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.802570 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:25.809274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.805900 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-b93a15-serving-cert\"" Oct 31 03:17:25.852656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.851938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:25.865674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.860640 4556 scope.go:122] "RemoveContainer" containerID="e82aa974704567cafb2e8736eaccfc68ea35f08b55882c5fa509ca15acb14958" Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.948401 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.948801 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.949016 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.949049 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmzk\" (UniqueName: \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.950165 4556 generic.go:356] "Generic (PLEG): container finished" podID="196598da-e0d8-4686-8ba0-069617cd19ad" containerID="aa855de6d01c7edf4fd06a599a8416b52034c6b69bfa419eb41f951bc2172d44" exitCode=0 Oct 31 03:17:25.952994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.950344 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" event={"ID":"196598da-e0d8-4686-8ba0-069617cd19ad","Type":"ContainerDied","Data":"aa855de6d01c7edf4fd06a599a8416b52034c6b69bfa419eb41f951bc2172d44"} Oct 31 03:17:25.982094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.971730 4556 scope.go:122] "RemoveContainer" containerID="fdf46e4779dd62afe140420845542eebecc64a5a8e7f11824f61905c46b755d9" Oct 31 03:17:25.998832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:25.998771 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:26.006599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.004149 4556 scope.go:122] "RemoveContainer" containerID="045856e80ce5428ccdec2811c5aac79092e71b1bfb9dd64e9692aa7868c50d2b" Oct 31 03:17:26.048249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.048221 4556 scope.go:122] "RemoveContainer" containerID="8f49bf615eb0d41cbc1ab97c00d9ae83bc03300ddda68837f33d80d705d6dd48" Oct 31 03:17:26.050910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.050880 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.051048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.050925 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmzk\" (UniqueName: \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.051486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.051281 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.051486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.051309 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.054747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.054714 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.059118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.055597 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.059118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.057256 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.062390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.062366 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmzk\" (UniqueName: \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\") pod \"maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.107790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.101544 4556 scope.go:122] "RemoveContainer" containerID="2d2ccc03ad3755990f9700b40ee8917d2d1578846d1227994efc0613ae2687e9" Oct 31 03:17:26.121193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.121161 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:17:26.124785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.124760 4556 scope.go:122] "RemoveContainer" containerID="d46d97c16d3491124dd3fb990523c540ac90fb692c35284444712e276e9ae785" Oct 31 03:17:26.130627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.130599 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:26.169691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.169659 4556 scope.go:122] "RemoveContainer" containerID="478ee0c8efcc55f69f27f203d739957490235147cc054ffaefed51098b370508" Oct 31 03:17:26.204339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.204022 4556 scope.go:122] "RemoveContainer" containerID="5b3f03b362ddc9be82aeea1a0db84af81ec750887ee6f7532a674dec6bee81e5" Oct 31 03:17:26.244186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.244149 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:26.245516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.244616 4556 scope.go:122] "RemoveContainer" containerID="04a9b25ff1c8dc3550a68e71807a85e752079efd60724cb98a0397e4172ea848" Oct 31 03:17:26.285513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.284424 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\") pod \"196598da-e0d8-4686-8ba0-069617cd19ad\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " Oct 31 03:17:26.285513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.284577 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\") pod \"196598da-e0d8-4686-8ba0-069617cd19ad\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " Oct 31 03:17:26.285513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.284607 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\") pod \"196598da-e0d8-4686-8ba0-069617cd19ad\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " Oct 31 03:17:26.285513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.284763 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\" (UniqueName: \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\") pod \"196598da-e0d8-4686-8ba0-069617cd19ad\" (UID: \"196598da-e0d8-4686-8ba0-069617cd19ad\") " Oct 31 03:17:26.294394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.294358 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca" pod "196598da-e0d8-4686-8ba0-069617cd19ad" (UID: "196598da-e0d8-4686-8ba0-069617cd19ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:26.299619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.299587 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/196598da-e0d8-4686-8ba0-069617cd19ad-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:26.309542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.308390 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls" pod "196598da-e0d8-4686-8ba0-069617cd19ad" (UID: "196598da-e0d8-4686-8ba0-069617cd19ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:26.309542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.308647 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs" pod "196598da-e0d8-4686-8ba0-069617cd19ad" (UID: "196598da-e0d8-4686-8ba0-069617cd19ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:26.309854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.309730 4556 scope.go:122] "RemoveContainer" containerID="5c8e38377ea5ec90ba78bc2c5b4ac7f760c46a065267e4ac01ac5e4b7987ab3c" Oct 31 03:17:26.315669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.315018 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq" pod "196598da-e0d8-4686-8ba0-069617cd19ad" (UID: "196598da-e0d8-4686-8ba0-069617cd19ad"). InnerVolumeSpecName "kube-api-access-xcksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:26.403744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.403709 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:26.403744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.403741 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcksq\" (UniqueName: \"kubernetes.io/projected/196598da-e0d8-4686-8ba0-069617cd19ad-kube-api-access-xcksq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:26.403744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.403752 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/196598da-e0d8-4686-8ba0-069617cd19ad-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:26.432310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.411890 4556 scope.go:122] "RemoveContainer" containerID="3ea982a36a0294343d3e70039808f2451817a061de0957b264b00a79a7fc1359" Oct 31 03:17:26.542382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.542344 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:26.737826 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:26.717848 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0349766b_07a9_47c1_b1ea_66971f8354b7.slice/crio-968d11e73662752d5ac6d7e847df3119472736adb79d793601cb3619918dbf9f WatchSource:0}: Error finding container 968d11e73662752d5ac6d7e847df3119472736adb79d793601cb3619918dbf9f: Status 404 returned error can't find the container with id 968d11e73662752d5ac6d7e847df3119472736adb79d793601cb3619918dbf9f Oct 31 03:17:26.737826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.719580 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:17:26.992399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.992355 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" event={"ID":"0349766b-07a9-47c1-b1ea-66971f8354b7","Type":"ContainerStarted","Data":"968d11e73662752d5ac6d7e847df3119472736adb79d793601cb3619918dbf9f"} Oct 31 03:17:26.997805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.997769 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" event={"ID":"196598da-e0d8-4686-8ba0-069617cd19ad","Type":"ContainerDied","Data":"5eb03fff6e1529b07f267d1d6303198a31885ad8f1f898a403c5ad5b8c817329"} Oct 31 03:17:26.997947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.997845 4556 scope.go:122] "RemoveContainer" containerID="aa855de6d01c7edf4fd06a599a8416b52034c6b69bfa419eb41f951bc2172d44" Oct 31 03:17:26.997947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:26.997781 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh" Oct 31 03:17:27.021589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:27.021528 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:17:27.027502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:27.027408 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1bfbe55-7b8cc59fcb-qhcmh"] Oct 31 03:17:27.700043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:27.699990 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:28.262280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:28.262190 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196598da-e0d8-4686-8ba0-069617cd19ad" path="/var/lib/kubelet/pods/196598da-e0d8-4686-8ba0-069617cd19ad/volumes" Oct 31 03:17:28.942686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:28.942591 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:28.951259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:28.951218 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:17:29.043782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.026358 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" event={"ID":"0349766b-07a9-47c1-b1ea-66971f8354b7","Type":"ContainerStarted","Data":"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303"} Oct 31 03:17:29.043782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.026468 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:29.178070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.049117 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" podStartSLOduration=2.906558045 podStartE2EDuration="4.049102447s" totalImagesPullingTime="1.142544402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:29.045092658 +0000 UTC m=+1326.900109799" watchObservedRunningTime="2026-10-31 03:17:29.049102447 +0000 UTC m=+1326.904119587" Oct 31 03:17:29.178070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.174397 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:17:29.178070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.174705 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" containerID="cri-o://a4e3a13d4e734696c9e35fb38d567d83a36b3dfd6247415522255438cb12dbb0" gracePeriod=30 Oct 31 03:17:29.197720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.197631 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:17:29.197720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.197719 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:17:29.197949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.197921 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" containerID="cri-o://896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018" gracePeriod=30 Oct 31 03:17:29.198213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:29.198188 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" containerID="cri-o://6955428a78ffcf1072fdabc05802bd2a02b123f9306975a8ea59a2006861cbbb" gracePeriod=30 Oct 31 03:17:30.044980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.044936 4556 generic.go:356] "Generic (PLEG): container finished" podID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerID="3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268" exitCode=0 Oct 31 03:17:30.045551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.045014 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerDied","Data":"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268"} Oct 31 03:17:30.045823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.045798 4556 scope.go:122] "RemoveContainer" containerID="3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268" Oct 31 03:17:30.048854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.048819 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:30.048987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.048894 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:30.048987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.048932 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:30.760297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.760242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:30.909258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:30.909215 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:31.065846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.065765 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerStarted","Data":"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273"} Oct 31 03:17:31.502437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.501884 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:17:31.502437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.502041 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:31.502437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.502083 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:31.518300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.518259 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:17:31.618735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.618688 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:31.802922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:31.802820 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:32.494465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:32.494429 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:32.494884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:32.494487 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:32.638075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:32.638038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:33.494849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:33.494471 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:33.494849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:33.494540 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:33.700869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:33.700828 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:33.912481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:33.912441 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.039605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.039573 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.301516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.301439 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.317349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.317315 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:17:34.318546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.318523 4556 cpu_manager.go:414] "containerMap: removing container" podUID="196598da-e0d8-4686-8ba0-069617cd19ad" containerName="maas-api" Oct 31 03:17:34.318546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.318547 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196598da-e0d8-4686-8ba0-069617cd19ad" containerName="maas-api" Oct 31 03:17:34.318740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.318727 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="196598da-e0d8-4686-8ba0-069617cd19ad" containerName="maas-api" Oct 31 03:17:34.326831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.326805 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.330131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.330091 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f1bfbe55\"" Oct 31 03:17:34.336016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.335935 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:17:34.341050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.340802 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:17:34.350664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.349651 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:17:34.350664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.349749 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.352649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.352486 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f1bfbe55-dockercfg-6d4t5\"" Oct 31 03:17:34.422433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.421186 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8mg\" (UniqueName: \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.422433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.421275 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.470292 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.511841 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.511913 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.523092 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8mg\" (UniqueName: \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.523173 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.523276 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.529598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.523484 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84972\" (UniqueName: \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.531494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.531388 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.759438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.758393 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.782635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.776539 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.782635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.776859 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84972\" (UniqueName: \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:34.804773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.804741 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8mg\" (UniqueName: \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\") pod \"payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:34.911739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:34.911699 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:35.028026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.027973 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84972\" (UniqueName: \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\") pod \"payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:35.112846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.111707 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:35.121227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.116543 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:17:35.328456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.328400 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:35.347035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.345491 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:17:35.394200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.394166 4556 generic.go:356] "Generic (PLEG): container finished" podID="937e3608-09f9-41bc-9b8b-610e75b72540" containerID="6955428a78ffcf1072fdabc05802bd2a02b123f9306975a8ea59a2006861cbbb" exitCode=0 Oct 31 03:17:35.394981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.394952 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" event={"ID":"937e3608-09f9-41bc-9b8b-610e75b72540","Type":"ContainerDied","Data":"6955428a78ffcf1072fdabc05802bd2a02b123f9306975a8ea59a2006861cbbb"} Oct 31 03:17:35.399464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.398776 4556 generic.go:356] "Generic (PLEG): container finished" podID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerID="896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018" exitCode=0 Oct 31 03:17:35.399464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.398880 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" event={"ID":"f87f39e5-7c66-4bfe-ba30-8ebf168d5425","Type":"ContainerDied","Data":"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018"} Oct 31 03:17:35.399464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.398924 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" event={"ID":"f87f39e5-7c66-4bfe-ba30-8ebf168d5425","Type":"ContainerDied","Data":"958ea14869c598ecca4a929aadab3d928618fe243d815323b2776e511a507722"} Oct 31 03:17:35.399464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.398950 4556 scope.go:122] "RemoveContainer" containerID="896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018" Oct 31 03:17:35.399464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.399105 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf" Oct 31 03:17:35.465059 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.464993 4556 scope.go:122] "RemoveContainer" containerID="896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018" Oct 31 03:17:35.465622 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:35.465595 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018\": container with ID starting with 896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018 not found: ID does not exist" containerID="896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018" Oct 31 03:17:35.465796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.465771 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018"} err="failed to get container status \"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018\": rpc error: code = NotFound desc = could not find container \"896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018\": container with ID starting with 896151668a1f25f220747827d8bb53afc28e00f856d48b24e539c3b35ef45018 not found: ID does not exist" Oct 31 03:17:35.466169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.466150 4556 generic.go:356] "Generic (PLEG): container finished" podID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerID="a4e3a13d4e734696c9e35fb38d567d83a36b3dfd6247415522255438cb12dbb0" exitCode=0 Oct 31 03:17:35.466585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.466567 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" event={"ID":"ac189756-7a86-45f8-aa2d-7f0f7399f4e3","Type":"ContainerDied","Data":"a4e3a13d4e734696c9e35fb38d567d83a36b3dfd6247415522255438cb12dbb0"} Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.503886 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504061 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504166 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504235 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504273 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504303 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504312 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504436 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504587 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504651 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\") pod \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\" (UID: \"f87f39e5-7c66-4bfe-ba30-8ebf168d5425\") " Oct 31 03:17:35.505056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504870 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.506067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.505850 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.506067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.505878 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.506067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.505880 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:35.506067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.505941 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:35.506359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.504873 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.507701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.507678 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.509131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.509104 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.512766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.512083 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:17:35.512766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.512483 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.516739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.516716 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.518169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.518055 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "kube-api-access-bjn5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.522027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.522005 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:17:35.526756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.526725 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy" pod "f87f39e5-7c66-4bfe-ba30-8ebf168d5425" (UID: "f87f39e5-7c66-4bfe-ba30-8ebf168d5425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.606961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.606918 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607021 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607062 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607100 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607130 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607167 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607195 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607253 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607298 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.607455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607328 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607464 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607561 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607646 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607709 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607758 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607769 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607797 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607849 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607858 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\") pod \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\" (UID: \"ac189756-7a86-45f8-aa2d-7f0f7399f4e3\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.607932 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\") pod \"937e3608-09f9-41bc-9b8b-610e75b72540\" (UID: \"937e3608-09f9-41bc-9b8b-610e75b72540\") " Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.608478 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.608693 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.609104 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.609554 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:35.610041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.609831 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.610066 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.610254 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611052 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611608 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611632 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611647 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611660 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611683 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611697 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611711 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611724 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611737 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611751 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611765 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611778 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937e3608-09f9-41bc-9b8b-610e75b72540-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611791 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjn5c\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-kube-api-access-bjn5c\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611804 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611817 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611830 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f87f39e5-7c66-4bfe-ba30-8ebf168d5425-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611844 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.612288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.611910 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.616291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.616262 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.616999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.616961 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.617869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.617841 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "kube-api-access-knkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.618068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.618025 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy" pod "ac189756-7a86-45f8-aa2d-7f0f7399f4e3" (UID: "ac189756-7a86-45f8-aa2d-7f0f7399f4e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:35.618068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.618030 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "kube-api-access-jhbxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.618068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.618051 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:35.624318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.624290 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo" pod "937e3608-09f9-41bc-9b8b-610e75b72540" (UID: "937e3608-09f9-41bc-9b8b-610e75b72540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:35.720905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.720869 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhbxd\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-kube-api-access-jhbxd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721068 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937e3608-09f9-41bc-9b8b-610e75b72540-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721088 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knkjg\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-kube-api-access-knkjg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721106 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937e3608-09f9-41bc-9b8b-610e75b72540-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721115 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937e3608-09f9-41bc-9b8b-610e75b72540-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721124 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721134 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.721365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.721143 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac189756-7a86-45f8-aa2d-7f0f7399f4e3-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:35.760923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.760861 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:17:35.765776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.765749 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-jjqvf"] Oct 31 03:17:35.805653 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:35.805604 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582f8ae2_93b2_4a89_8cf9_cf6d922800c8.slice/crio-5e38890680c537699ed2d6b77f940ac781f3f4cbcee92aa16eec0b1b8d9db716 WatchSource:0}: Error finding container 5e38890680c537699ed2d6b77f940ac781f3f4cbcee92aa16eec0b1b8d9db716: Status 404 returned error can't find the container with id 5e38890680c537699ed2d6b77f940ac781f3f4cbcee92aa16eec0b1b8d9db716 Oct 31 03:17:35.806148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.806124 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:17:35.817922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:35.817892 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:17:36.299176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.296260 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:36.360150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.360115 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" path="/var/lib/kubelet/pods/f87f39e5-7c66-4bfe-ba30-8ebf168d5425/volumes" Oct 31 03:17:36.494638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.494591 4556 patch_prober.go:29] interesting pod/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 31 03:17:36.494974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.494949 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 31 03:17:36.503338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.503315 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" Oct 31 03:17:36.503596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.503352 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt" event={"ID":"ac189756-7a86-45f8-aa2d-7f0f7399f4e3","Type":"ContainerDied","Data":"a77a3387fed66b45b221e85213877c295d3dfd1d78ee8deca06a196456388d15"} Oct 31 03:17:36.503687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.503663 4556 scope.go:122] "RemoveContainer" containerID="a4e3a13d4e734696c9e35fb38d567d83a36b3dfd6247415522255438cb12dbb0" Oct 31 03:17:36.506595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.505833 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" event={"ID":"582f8ae2-93b2-4a89-8cf9-cf6d922800c8","Type":"ContainerStarted","Data":"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9"} Oct 31 03:17:36.506595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.505888 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" event={"ID":"582f8ae2-93b2-4a89-8cf9-cf6d922800c8","Type":"ContainerStarted","Data":"5e38890680c537699ed2d6b77f940ac781f3f4cbcee92aa16eec0b1b8d9db716"} Oct 31 03:17:36.506967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.506861 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:36.509150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.509095 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" event={"ID":"b68b57e0-574c-43b7-8b6c-40fa82c1d41a","Type":"ContainerStarted","Data":"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64"} Oct 31 03:17:36.509150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.509139 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" event={"ID":"b68b57e0-574c-43b7-8b6c-40fa82c1d41a","Type":"ContainerStarted","Data":"318147da9a01fb80cc98100c3f997218b83b005d05a392dd7ab1f803c2fbd6d4"} Oct 31 03:17:36.509347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.509214 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:36.512642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.511154 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" event={"ID":"937e3608-09f9-41bc-9b8b-610e75b72540","Type":"ContainerDied","Data":"901b7f2e77ab87d8503c678d96b4715761d457a481254102a9567231e390cf36"} Oct 31 03:17:36.512642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.511232 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd" Oct 31 03:17:36.529137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.529020 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" podStartSLOduration=2.529002925 podStartE2EDuration="2.529002925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:36.524516421 +0000 UTC m=+1334.379533561" watchObservedRunningTime="2026-10-31 03:17:36.529002925 +0000 UTC m=+1334.384020066" Oct 31 03:17:36.547454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.546425 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:17:36.551564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.551532 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-m6xqt"] Oct 31 03:17:36.558210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.558173 4556 scope.go:122] "RemoveContainer" containerID="6955428a78ffcf1072fdabc05802bd2a02b123f9306975a8ea59a2006861cbbb" Oct 31 03:17:36.566948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.564667 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:17:36.598461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.598156 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" podStartSLOduration=2.598134178 podStartE2EDuration="2.598134178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:36.583088175 +0000 UTC m=+1334.438105318" watchObservedRunningTime="2026-10-31 03:17:36.598134178 +0000 UTC m=+1334.453151321" Oct 31 03:17:36.599348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.599325 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-cnmcd"] Oct 31 03:17:36.959117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:36.954180 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:17:37.044770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.044727 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:37.185339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.185301 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:37.503439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.502786 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:17:37.507436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.504599 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:17:37.507436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.506836 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:17:37.711963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:37.711918 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:38.260564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:38.260530 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" path="/var/lib/kubelet/pods/937e3608-09f9-41bc-9b8b-610e75b72540/volumes" Oct 31 03:17:38.261228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:38.261210 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" path="/var/lib/kubelet/pods/ac189756-7a86-45f8-aa2d-7f0f7399f4e3/volumes" Oct 31 03:17:38.363616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:38.363569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:38.604970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:38.604877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:39.349437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:39.343242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:40.045691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:40.045654 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:40.742106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:40.742070 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:41.433251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:41.433213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:41.955350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:41.955301 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" containerID="cri-o://79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8" gracePeriod=25 Oct 31 03:17:42.147560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:42.147008 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:42.807269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:42.807236 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:42.955439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:42.950358 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:43.223640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.223606 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:43.358354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.358317 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:43.964604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.964554 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:17:43.965011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.964885 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" containerID="cri-o://c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42" gracePeriod=30 Oct 31 03:17:43.971053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.971024 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:17:43.971214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:43.971195 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" containerID="cri-o://32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150" gracePeriod=30 Oct 31 03:17:44.171393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:44.171348 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:44.757698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:44.757651 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:17:44.757963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:44.757912 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" podUID="1fb63b71-f423-4052-9756-4e7258aa6a92" containerName="maas-api" containerID="cri-o://7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393" gracePeriod=30 Oct 31 03:17:44.820759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:44.820727 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.260554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.253682 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.383259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.383225 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:17:45.464645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.464611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.478171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.478136 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\" (UniqueName: \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\") pod \"1fb63b71-f423-4052-9756-4e7258aa6a92\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " Oct 31 03:17:45.478328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.478213 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\") pod \"1fb63b71-f423-4052-9756-4e7258aa6a92\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " Oct 31 03:17:45.478328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.478322 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\") pod \"1fb63b71-f423-4052-9756-4e7258aa6a92\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " Oct 31 03:17:45.478521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.478395 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\") pod \"1fb63b71-f423-4052-9756-4e7258aa6a92\" (UID: \"1fb63b71-f423-4052-9756-4e7258aa6a92\") " Oct 31 03:17:45.479096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.479063 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca" pod "1fb63b71-f423-4052-9756-4e7258aa6a92" (UID: "1fb63b71-f423-4052-9756-4e7258aa6a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:45.479425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.479386 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb63b71-f423-4052-9756-4e7258aa6a92-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:45.481264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.481241 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx" pod "1fb63b71-f423-4052-9756-4e7258aa6a92" (UID: "1fb63b71-f423-4052-9756-4e7258aa6a92"). InnerVolumeSpecName "kube-api-access-9jrbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:45.481352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.481269 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls" pod "1fb63b71-f423-4052-9756-4e7258aa6a92" (UID: "1fb63b71-f423-4052-9756-4e7258aa6a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:45.481931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.481905 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs" pod "1fb63b71-f423-4052-9756-4e7258aa6a92" (UID: "1fb63b71-f423-4052-9756-4e7258aa6a92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:45.581937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.581847 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrbx\" (UniqueName: \"kubernetes.io/projected/1fb63b71-f423-4052-9756-4e7258aa6a92-kube-api-access-9jrbx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:45.581937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.581890 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:45.581937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.581901 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb63b71-f423-4052-9756-4e7258aa6a92-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:45.587166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.587127 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.650749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.650710 4556 generic.go:356] "Generic (PLEG): container finished" podID="1fb63b71-f423-4052-9756-4e7258aa6a92" containerID="7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393" exitCode=0 Oct 31 03:17:45.650992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.650795 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" Oct 31 03:17:45.650992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.650941 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" event={"ID":"1fb63b71-f423-4052-9756-4e7258aa6a92","Type":"ContainerDied","Data":"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393"} Oct 31 03:17:45.650992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.650988 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd" event={"ID":"1fb63b71-f423-4052-9756-4e7258aa6a92","Type":"ContainerDied","Data":"9e265ac6ea8d15cf3f243a33e10b6ea9ddb1e5162034e19d970471f5b7a12613"} Oct 31 03:17:45.651234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.651018 4556 scope.go:122] "RemoveContainer" containerID="7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393" Oct 31 03:17:45.668580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.668532 4556 scope.go:122] "RemoveContainer" containerID="7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393" Oct 31 03:17:45.669067 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:45.669045 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393\": container with ID starting with 7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393 not found: ID does not exist" containerID="7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393" Oct 31 03:17:45.669121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.669079 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393"} err="failed to get container status \"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393\": rpc error: code = NotFound desc = could not find container \"7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393\": container with ID starting with 7d1fb1f9bfd14f6db9ab53dea6a99dc17c483cf2bb83e7d5d363716d6fc03393 not found: ID does not exist" Oct 31 03:17:45.678799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.678758 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:17:45.683263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.683233 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2ad1f0-584f9946d9-4x2nd"] Oct 31 03:17:45.727273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.727228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.852923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:45.852827 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:45.962169 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:45.962127 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:17:46.067468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.067380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:46.206657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.206624 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:46.261031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.260997 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb63b71-f423-4052-9756-4e7258aa6a92" path="/var/lib/kubelet/pods/1fb63b71-f423-4052-9756-4e7258aa6a92/volumes" Oct 31 03:17:46.327715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.327665 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:17:46.328923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328897 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" Oct 31 03:17:46.329004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328927 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" Oct 31 03:17:46.329004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328945 4556 cpu_manager.go:414] "containerMap: removing container" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" Oct 31 03:17:46.329004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328953 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" Oct 31 03:17:46.329004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328980 4556 cpu_manager.go:414] "containerMap: removing container" podUID="1fb63b71-f423-4052-9756-4e7258aa6a92" containerName="maas-api" Oct 31 03:17:46.329004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.328988 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb63b71-f423-4052-9756-4e7258aa6a92" containerName="maas-api" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329034 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329043 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329255 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb63b71-f423-4052-9756-4e7258aa6a92" containerName="maas-api" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329279 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f87f39e5-7c66-4bfe-ba30-8ebf168d5425" containerName="istio-proxy" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329298 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac189756-7a86-45f8-aa2d-7f0f7399f4e3" containerName="istio-proxy" Oct 31 03:17:46.329865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.329307 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="937e3608-09f9-41bc-9b8b-610e75b72540" containerName="istio-proxy" Oct 31 03:17:46.334732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.334708 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.355028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.354998 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:17:46.379627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.379586 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-6d1a8e-openshift-default-dockercfg-bjjnf\"" Oct 31 03:17:46.397186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397146 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnlt\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.397337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397193 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.397444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397395 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.397615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397530 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.398362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397675 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.398362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397826 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.398362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397871 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.398362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397890 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.398362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.397972 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.410346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.410140 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:46.508039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.507944 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.511572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.508354 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.511572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.508465 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.511572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.508494 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.511572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.508529 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.519436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.517244 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.529955 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnlt\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.530013 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.530175 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.530232 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.530948 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.531172 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.532334 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.534437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.532682 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.556510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.555035 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.600439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.594153 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:46.600439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.595326 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.600439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.595784 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.600439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.599088 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnlt\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\") pod \"e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.664441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.649783 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:46.697034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:46.692658 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" containerID="cri-o://25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810" gracePeriod=45 Oct 31 03:17:47.073344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.073315 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:17:47.083461 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:47.083383 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34191e62_c455_40b4_83cb_5d7a2b3a6b85.slice/crio-27fc67904cae0126f350b5800b2a6c5cabf98a7f8f49db1c7321eed7b837dfca WatchSource:0}: Error finding container 27fc67904cae0126f350b5800b2a6c5cabf98a7f8f49db1c7321eed7b837dfca: Status 404 returned error can't find the container with id 27fc67904cae0126f350b5800b2a6c5cabf98a7f8f49db1c7321eed7b837dfca Oct 31 03:17:47.083575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.083483 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:17:47.086864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.086825 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:47.086953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.086917 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:47.086993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.086963 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:17:47.164319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.164286 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\" (UniqueName: \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.164912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.164858 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.164912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.164905 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.165110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.164980 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.165385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.165362 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.165522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.165429 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\") pod \"51cef924-2747-43a9-a93a-f40d70e3cdaf\" (UID: \"51cef924-2747-43a9-a93a-f40d70e3cdaf\") " Oct 31 03:17:47.166649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.166330 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.167227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.167203 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.167482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.167270 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.168066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.168031 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "kube-api-access-vq8vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:47.169330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.169307 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:47.169458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.169329 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs" pod "51cef924-2747-43a9-a93a-f40d70e3cdaf" (UID: "51cef924-2747-43a9-a93a-f40d70e3cdaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268811 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268845 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268861 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8vd\" (UniqueName: \"kubernetes.io/projected/51cef924-2747-43a9-a93a-f40d70e3cdaf-kube-api-access-vq8vd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268879 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cef924-2747-43a9-a93a-f40d70e3cdaf-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268893 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.268937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.268905 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cef924-2747-43a9-a93a-f40d70e3cdaf-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:47.340349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.340261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:47.560569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.560542 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:17:47.785569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.785525 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerStarted","Data":"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5"} Oct 31 03:17:47.785787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.785597 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerStarted","Data":"27fc67904cae0126f350b5800b2a6c5cabf98a7f8f49db1c7321eed7b837dfca"} Oct 31 03:17:47.789044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.789013 4556 generic.go:356] "Generic (PLEG): container finished" podID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerID="25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810" exitCode=0 Oct 31 03:17:47.789182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.789052 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" Oct 31 03:17:47.789182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.789071 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" event={"ID":"51cef924-2747-43a9-a93a-f40d70e3cdaf","Type":"ContainerDied","Data":"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810"} Oct 31 03:17:47.789182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.789111 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz" event={"ID":"51cef924-2747-43a9-a93a-f40d70e3cdaf","Type":"ContainerDied","Data":"4092732b6276e192a086ec2ec33e9fdac6a210d4f03fabfebcd60ef88066aefb"} Oct 31 03:17:47.789182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.789136 4556 scope.go:122] "RemoveContainer" containerID="25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810" Oct 31 03:17:47.805065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.805041 4556 scope.go:122] "RemoveContainer" containerID="25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810" Oct 31 03:17:47.805458 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:47.805423 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810\": container with ID starting with 25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810 not found: ID does not exist" containerID="25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810" Oct 31 03:17:47.805570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.805469 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810"} err="failed to get container status \"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810\": rpc error: code = NotFound desc = could not find container \"25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810\": container with ID starting with 25c182f687c63da9e19121eff848401473728eff36381684475e035ac0edb810 not found: ID does not exist" Oct 31 03:17:47.809580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.809522 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" podStartSLOduration=1.809505718 podStartE2EDuration="1.809505718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:47.808489484 +0000 UTC m=+1345.663506622" watchObservedRunningTime="2026-10-31 03:17:47.809505718 +0000 UTC m=+1345.664522860" Oct 31 03:17:47.832531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.832490 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:17:47.836847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:47.836817 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-56afbd/simulator-w5-56afbd-kserve-f88776d94-nh6mz"] Oct 31 03:17:48.260883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.260846 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" path="/var/lib/kubelet/pods/51cef924-2747-43a9-a93a-f40d70e3cdaf/volumes" Oct 31 03:17:48.387433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.387271 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:48.565399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.565308 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:17:48.665347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.665312 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:48.671701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.671663 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:48.799959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.799924 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:48.801093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:48.801068 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:17:49.281505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.281473 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:17:49.338810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.338774 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:17:49.421171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421126 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421181 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421258 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421286 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421317 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421363 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421467 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421499 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421574 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421621 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421692 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421748 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421819 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.421930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421903 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.422106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.421957 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.422106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.422023 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\") pod \"92e097c0-a066-404d-ba85-709caa6abfb6\" (UID: \"92e097c0-a066-404d-ba85-709caa6abfb6\") " Oct 31 03:17:49.422106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.422092 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.422210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.422198 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\") pod \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\" (UID: \"a4b56b41-c5b0-4154-b123-b7d113bb9f6a\") " Oct 31 03:17:49.422754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.422725 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.423823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.423798 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:49.424029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.423941 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.424202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.424153 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.424556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.424535 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.424910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.424864 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:49.425336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.425239 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:49.425572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.425552 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.425906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.425885 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.426231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.426209 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.426581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.426559 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.429968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.429937 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:49.431755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.431728 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:17:49.433845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.433813 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.433939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.433851 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "kube-api-access-gbkrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:49.433939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.433886 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:49.434381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.434354 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "kube-api-access-42rq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:49.434523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.434498 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy" pod "a4b56b41-c5b0-4154-b123-b7d113bb9f6a" (UID: "a4b56b41-c5b0-4154-b123-b7d113bb9f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:49.435162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.435136 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token" pod "92e097c0-a066-404d-ba85-709caa6abfb6" (UID: "92e097c0-a066-404d-ba85-709caa6abfb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524709 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e097c0-a066-404d-ba85-709caa6abfb6-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524744 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524758 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524767 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524778 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524788 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rq9\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-kube-api-access-42rq9\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524797 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524805 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524815 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524823 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524831 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524839 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkrl\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-kube-api-access-gbkrl\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524849 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.524850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524857 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.525461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524867 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e097c0-a066-404d-ba85-709caa6abfb6-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.525461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524874 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e097c0-a066-404d-ba85-709caa6abfb6-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.525461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524882 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4b56b41-c5b0-4154-b123-b7d113bb9f6a-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.525461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.524891 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e097c0-a066-404d-ba85-709caa6abfb6-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:49.810358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.810256 4556 generic.go:356] "Generic (PLEG): container finished" podID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerID="32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150" exitCode=0 Oct 31 03:17:49.810358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.810308 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" Oct 31 03:17:49.810358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.810343 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" event={"ID":"a4b56b41-c5b0-4154-b123-b7d113bb9f6a","Type":"ContainerDied","Data":"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150"} Oct 31 03:17:49.810670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.810398 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5" event={"ID":"a4b56b41-c5b0-4154-b123-b7d113bb9f6a","Type":"ContainerDied","Data":"6d46576552114a471f2edca30a73115a21a6a0543223136289df17e65cdcc14c"} Oct 31 03:17:49.810670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.810437 4556 scope.go:122] "RemoveContainer" containerID="32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150" Oct 31 03:17:49.812087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.812052 4556 generic.go:356] "Generic (PLEG): container finished" podID="92e097c0-a066-404d-ba85-709caa6abfb6" containerID="c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42" exitCode=0 Oct 31 03:17:49.812228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.812112 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" Oct 31 03:17:49.812228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.812140 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" event={"ID":"92e097c0-a066-404d-ba85-709caa6abfb6","Type":"ContainerDied","Data":"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42"} Oct 31 03:17:49.812228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.812200 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp" event={"ID":"92e097c0-a066-404d-ba85-709caa6abfb6","Type":"ContainerDied","Data":"ef22ca57075544aeaad63e9b62c2558e89a28bd996cc1fad2ce872148c987e13"} Oct 31 03:17:49.827615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.827586 4556 scope.go:122] "RemoveContainer" containerID="32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150" Oct 31 03:17:49.828014 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:49.827990 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150\": container with ID starting with 32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150 not found: ID does not exist" containerID="32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150" Oct 31 03:17:49.828119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.828024 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150"} err="failed to get container status \"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150\": rpc error: code = NotFound desc = could not find container \"32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150\": container with ID starting with 32b1f5a11d6cc046b7536cdb5b78f29001e306aa6a38719f873dfe4cea287150 not found: ID does not exist" Oct 31 03:17:49.828119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.828043 4556 scope.go:122] "RemoveContainer" containerID="c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42" Oct 31 03:17:49.842870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.841471 4556 scope.go:122] "RemoveContainer" containerID="c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42" Oct 31 03:17:49.842870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.841812 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:17:49.843234 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:17:49.843199 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42\": container with ID starting with c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42 not found: ID does not exist" containerID="c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42" Oct 31 03:17:49.843347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.843246 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42"} err="failed to get container status \"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42\": rpc error: code = NotFound desc = could not find container \"c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42\": container with ID starting with c492771352e676243bbce603172002d67d3c3b7a57c0e5d52c8649030bf82c42 not found: ID does not exist" Oct 31 03:17:49.847711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.847688 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-5rkh5"] Oct 31 03:17:49.856709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.856674 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:17:49.861151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.861129 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-n6kmp"] Oct 31 03:17:49.966409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:49.966369 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:50.233872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:50.233836 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:50.284293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:50.279275 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" path="/var/lib/kubelet/pods/92e097c0-a066-404d-ba85-709caa6abfb6/volumes" Oct 31 03:17:50.284293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:50.280206 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" path="/var/lib/kubelet/pods/a4b56b41-c5b0-4154-b123-b7d113bb9f6a/volumes" Oct 31 03:17:50.440847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:50.440797 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:50.894693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:50.894648 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:17:51.194669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.194625 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:51.469558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.469464 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:17:51.470321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470302 4556 cpu_manager.go:414] "containerMap: removing container" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" Oct 31 03:17:51.470321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470320 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470340 4556 cpu_manager.go:414] "containerMap: removing container" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470347 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470363 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470368 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470508 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cef924-2747-43a9-a93a-f40d70e3cdaf" containerName="main" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470520 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e097c0-a066-404d-ba85-709caa6abfb6" containerName="istio-proxy" Oct 31 03:17:51.470556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.470528 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b56b41-c5b0-4154-b123-b7d113bb9f6a" containerName="istio-proxy" Oct 31 03:17:51.476532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.476502 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.478341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.478289 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:51.482173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.480778 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6d1a8e\"" Oct 31 03:17:51.486554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.486511 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:17:51.487980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.487954 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x52c\" (UniqueName: \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.488117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.488029 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.512265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.512224 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:17:51.522803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.522767 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.526305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.526270 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:17:51.527501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.527367 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6d1a8e-dockercfg-6542x\"" Oct 31 03:17:51.616437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.596987 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.616437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.597143 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x52c\" (UniqueName: \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.616437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.597238 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.616437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.597331 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvz57\" (UniqueName: \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.616437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.603733 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.659020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.624521 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:51.659020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.626847 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x52c\" (UniqueName: \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\") pod \"payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.721526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.709771 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.721526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.710030 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvz57\" (UniqueName: \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.722251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.721802 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.739540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.730895 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvz57\" (UniqueName: \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\") pod \"payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:51.812438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.811102 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:51.848827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:51.846808 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:52.206469 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:52.205500 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e4d2b5_2050_4873_b69c_beafc19b9f0e.slice/crio-7ed73ebc7151ae1b4359c59d4a83d734592f22d8dc9ea60c2853579cd4c0c506 WatchSource:0}: Error finding container 7ed73ebc7151ae1b4359c59d4a83d734592f22d8dc9ea60c2853579cd4c0c506: Status 404 returned error can't find the container with id 7ed73ebc7151ae1b4359c59d4a83d734592f22d8dc9ea60c2853579cd4c0c506 Oct 31 03:17:52.206469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.206194 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:17:52.265535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.265485 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:17:52.332127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.332097 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:52.864658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.864619 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" event={"ID":"28b17cad-376e-4e85-8477-e153e058a644","Type":"ContainerStarted","Data":"0b53cbcb6815904a97a380d1aa71808d994afd4b5c7f8bab535b24842a5144d2"} Oct 31 03:17:52.865163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.864685 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" event={"ID":"28b17cad-376e-4e85-8477-e153e058a644","Type":"ContainerStarted","Data":"53ca1671b8c07b65ebae942c262f1fdfdced42063c9e7bd9bbb68ea4e26dc5e1"} Oct 31 03:17:52.865163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.864719 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:17:52.868464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.868427 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" event={"ID":"84e4d2b5-2050-4873-b69c-beafc19b9f0e","Type":"ContainerStarted","Data":"65537a73aba0194b1698904b93881d047eb4bc5d867ea0466411def835df7101"} Oct 31 03:17:52.868652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.868484 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" event={"ID":"84e4d2b5-2050-4873-b69c-beafc19b9f0e","Type":"ContainerStarted","Data":"7ed73ebc7151ae1b4359c59d4a83d734592f22d8dc9ea60c2853579cd4c0c506"} Oct 31 03:17:52.868652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.868596 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:17:52.886675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.886606 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" podStartSLOduration=1.886586324 podStartE2EDuration="1.886586324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:52.884349436 +0000 UTC m=+1350.739366582" watchObservedRunningTime="2026-10-31 03:17:52.886586324 +0000 UTC m=+1350.741603469" Oct 31 03:17:52.921161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:52.921003 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" podStartSLOduration=1.9209788589999999 podStartE2EDuration="1.920978859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:52.909004482 +0000 UTC m=+1350.764021622" watchObservedRunningTime="2026-10-31 03:17:52.920978859 +0000 UTC m=+1350.775996031" Oct 31 03:17:53.466730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:53.466639 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:54.197629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.197590 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4"] Oct 31 03:17:54.357123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.357083 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.365470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.364976 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:17:54.365470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.365120 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:17:54.409363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.395119 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4"] Oct 31 03:17:54.475445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.475328 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7vd\" (UniqueName: \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.477222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.477099 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.478037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.477830 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.603868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.602078 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7vd\" (UniqueName: \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.603868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.602383 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.603868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.603378 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.603868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.603786 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.636865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.636795 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.664275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.664234 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7vd\" (UniqueName: \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\") pod \"maas-api-revoke-sub-fb496ddc-7tkc4\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.715551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.715514 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:54.833748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:54.827902 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:55.037224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.037009 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:55.084156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.077980 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4"] Oct 31 03:17:55.208667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.208629 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:17:55.209142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.208849 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" containerID="cri-o://59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273" gracePeriod=30 Oct 31 03:17:55.253986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.253938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:55.812202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.812100 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87"] Oct 31 03:17:55.817073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.817051 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:55.826763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.826725 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87"] Oct 31 03:17:55.895544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.895437 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:55.895752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.895567 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:55.895752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.895613 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9d6\" (UniqueName: \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:55.934293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.934253 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" event={"ID":"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7","Type":"ContainerStarted","Data":"e09ab7de1a5c8232170baf1c84ad1bfb9cd30fa568d4bceda4a829edee193852"} Oct 31 03:17:55.934555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.934316 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" event={"ID":"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7","Type":"ContainerStarted","Data":"3cebdeba5f1457e5d6706286cb34ad540597e38ba2498867910b94f9473d1552"} Oct 31 03:17:55.951067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:55.951006 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" podStartSLOduration=1.950990178 podStartE2EDuration="1.950990178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:55.950801024 +0000 UTC m=+1353.805818178" watchObservedRunningTime="2026-10-31 03:17:55.950990178 +0000 UTC m=+1353.806007318" Oct 31 03:17:56.001460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.000226 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.001460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.000311 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.001460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.000353 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9d6\" (UniqueName: \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.002495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.002449 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.004959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.004864 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.013377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.013336 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9d6\" (UniqueName: \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\") pod \"maas-api-revoke-keys-e2e-api-b-780abd-75r87\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.136554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.136395 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:56.453646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.453604 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87"] Oct 31 03:17:56.454177 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:17:56.454150 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d11af69_591e_4878_9c5b_aa9f198100b4.slice/crio-8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107 WatchSource:0}: Error finding container 8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107: Status 404 returned error can't find the container with id 8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107 Oct 31 03:17:56.482344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.482255 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:56.946606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.946569 4556 generic.go:356] "Generic (PLEG): container finished" podID="0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" containerID="e09ab7de1a5c8232170baf1c84ad1bfb9cd30fa568d4bceda4a829edee193852" exitCode=0 Oct 31 03:17:56.946815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.946656 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" event={"ID":"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7","Type":"ContainerDied","Data":"e09ab7de1a5c8232170baf1c84ad1bfb9cd30fa568d4bceda4a829edee193852"} Oct 31 03:17:56.948263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:56.948237 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" event={"ID":"0d11af69-591e-4878-9c5b-aa9f198100b4","Type":"ContainerStarted","Data":"8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107"} Oct 31 03:17:57.300810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:57.300778 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:57.910880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:57.910845 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:57.959035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:57.958993 4556 generic.go:356] "Generic (PLEG): container finished" podID="0d11af69-591e-4878-9c5b-aa9f198100b4" containerID="9014270af38b59a3b5a5a8bd550cca99dcee2cdc5cc46d9b2d18b341887f4859" exitCode=0 Oct 31 03:17:57.959228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:57.959038 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" event={"ID":"0d11af69-591e-4878-9c5b-aa9f198100b4","Type":"ContainerDied","Data":"9014270af38b59a3b5a5a8bd550cca99dcee2cdc5cc46d9b2d18b341887f4859"} Oct 31 03:17:58.105498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.090397 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:58.312573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.305104 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:58.491440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.466944 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:58.622466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.620882 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:58.646346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.646304 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\") pod \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " Oct 31 03:17:58.646576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.646491 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\") pod \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " Oct 31 03:17:58.646576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.646544 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\" (UniqueName: \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\") pod \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\" (UID: \"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7\") " Oct 31 03:17:58.647659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.647597 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca" pod "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" (UID: "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:58.651105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.651056 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca" pod "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" (UID: "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:58.651245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.651180 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd" pod "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" (UID: "0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7"). InnerVolumeSpecName "kube-api-access-dn7vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:58.749358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.749269 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.749358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.749310 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.749358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.749324 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7vd\" (UniqueName: \"kubernetes.io/projected/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7-kube-api-access-dn7vd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:58.790448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.786710 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:58.997345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.997308 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" Oct 31 03:17:59.000370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.999533 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4" event={"ID":"0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7","Type":"ContainerDied","Data":"3cebdeba5f1457e5d6706286cb34ad540597e38ba2498867910b94f9473d1552"} Oct 31 03:17:59.000370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:58.999596 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cebdeba5f1457e5d6706286cb34ad540597e38ba2498867910b94f9473d1552" Oct 31 03:17:59.238152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.238118 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:17:59.259902 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.259798 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\") pod \"0d11af69-591e-4878-9c5b-aa9f198100b4\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " Oct 31 03:17:59.260128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.260021 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\") pod \"0d11af69-591e-4878-9c5b-aa9f198100b4\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " Oct 31 03:17:59.260128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.260117 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\" (UniqueName: \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\") pod \"0d11af69-591e-4878-9c5b-aa9f198100b4\" (UID: \"0d11af69-591e-4878-9c5b-aa9f198100b4\") " Oct 31 03:17:59.261123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.260811 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca" pod "0d11af69-591e-4878-9c5b-aa9f198100b4" (UID: "0d11af69-591e-4878-9c5b-aa9f198100b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:59.261326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.261304 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:59.262604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.262558 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca" pod "0d11af69-591e-4878-9c5b-aa9f198100b4" (UID: "0d11af69-591e-4878-9c5b-aa9f198100b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:59.262993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.262973 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6" pod "0d11af69-591e-4878-9c5b-aa9f198100b4" (UID: "0d11af69-591e-4878-9c5b-aa9f198100b4"). InnerVolumeSpecName "kube-api-access-vs9d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:59.363077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.363044 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d11af69-591e-4878-9c5b-aa9f198100b4-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:59.363077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.363072 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs9d6\" (UniqueName: \"kubernetes.io/projected/0d11af69-591e-4878-9c5b-aa9f198100b4-kube-api-access-vs9d6\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:59.401434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:17:59.399163 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:00.008886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.008858 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" Oct 31 03:18:00.009355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.008891 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87" event={"ID":"0d11af69-591e-4878-9c5b-aa9f198100b4","Type":"ContainerDied","Data":"8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107"} Oct 31 03:18:00.009355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.008947 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cfbf8724d3644bdd88bbac5faef9b89b10272973c604fdde4fbdd39c8b59107" Oct 31 03:18:00.297452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.292857 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:00.595698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.595648 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:18:00.722835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.722792 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.722894 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.722952 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723019 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723074 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723124 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723188 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723326 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723369 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\") pod \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\" (UID: \"b2ca4c51-2abf-4bf1-9706-08d321b33d5a\") " Oct 31 03:18:00.723514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723409 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:00.723958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723891 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:00.724077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723963 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:00.724077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.723902 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:00.724317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.724298 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:00.724447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.724406 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.724529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.724457 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.724529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.724472 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.724529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.724488 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.726037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.726004 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:00.726399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.726376 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:00.726871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.726833 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:00.727608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.727582 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8" pod "b2ca4c51-2abf-4bf1-9706-08d321b33d5a" (UID: "b2ca4c51-2abf-4bf1-9706-08d321b33d5a"). InnerVolumeSpecName "kube-api-access-6jmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:00.826591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.826495 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.826591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.826536 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.826591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.826549 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.826591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.826557 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.826591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.826567 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmq8\" (UniqueName: \"kubernetes.io/projected/b2ca4c51-2abf-4bf1-9706-08d321b33d5a-kube-api-access-6jmq8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:00.894520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:00.894482 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:18:01.018193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.018154 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:01.024012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.022942 4556 generic.go:356] "Generic (PLEG): container finished" podID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerID="59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273" exitCode=0 Oct 31 03:18:01.024012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.023341 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" Oct 31 03:18:01.035979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.023159 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerDied","Data":"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273"} Oct 31 03:18:01.035979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.031547 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc" event={"ID":"b2ca4c51-2abf-4bf1-9706-08d321b33d5a","Type":"ContainerDied","Data":"64e53b706d99d2ce42c208c7ad65a957667864e97a79dc15dd9ebd699327c14c"} Oct 31 03:18:01.035979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.031659 4556 scope.go:122] "RemoveContainer" containerID="59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273" Oct 31 03:18:01.071288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.071262 4556 scope.go:122] "RemoveContainer" containerID="3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268" Oct 31 03:18:01.079396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.079317 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:18:01.087751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.087635 4556 scope.go:122] "RemoveContainer" containerID="59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273" Oct 31 03:18:01.088214 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:01.088184 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273\": container with ID starting with 59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273 not found: ID does not exist" containerID="59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273" Oct 31 03:18:01.088550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.088227 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273"} err="failed to get container status \"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273\": rpc error: code = NotFound desc = could not find container \"59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273\": container with ID starting with 59bae792a7ecd50e76a4befe4f6e5c22b792f42225fa36d71fd9120b876ec273 not found: ID does not exist" Oct 31 03:18:01.088550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.088256 4556 scope.go:122] "RemoveContainer" containerID="3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268" Oct 31 03:18:01.095231 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:01.094748 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268\": container with ID starting with 3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268 not found: ID does not exist" containerID="3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268" Oct 31 03:18:01.095231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.094800 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268"} err="failed to get container status \"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268\": rpc error: code = NotFound desc = could not find container \"3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268\": container with ID starting with 3a1501060d8e7e6e389a8bf6c368c31fd92a1f333804fc84bd0f23e76c437268 not found: ID does not exist" Oct 31 03:18:01.095548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.095289 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2ad1f0-openshift-default-7f885957df-bw7wc"] Oct 31 03:18:01.594086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:01.594050 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:02.094123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.094084 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:02.276523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.273635 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" path="/var/lib/kubelet/pods/b2ca4c51-2abf-4bf1-9706-08d321b33d5a/volumes" Oct 31 03:18:02.337505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.337475 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:18:02.426922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.426876 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\") pod \"76188f96-56cb-4c55-b324-205cacde847c\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " Oct 31 03:18:02.427302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.427283 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\" (UniqueName: \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\") pod \"76188f96-56cb-4c55-b324-205cacde847c\" (UID: \"76188f96-56cb-4c55-b324-205cacde847c\") " Oct 31 03:18:02.431360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.431322 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume" pod "76188f96-56cb-4c55-b324-205cacde847c" (UID: "76188f96-56cb-4c55-b324-205cacde847c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:02.432594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.432551 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg" pod "76188f96-56cb-4c55-b324-205cacde847c" (UID: "76188f96-56cb-4c55-b324-205cacde847c"). InnerVolumeSpecName "kube-api-access-vqjlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:02.530364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.530260 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76188f96-56cb-4c55-b324-205cacde847c-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:02.530364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.530305 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqjlg\" (UniqueName: \"kubernetes.io/projected/76188f96-56cb-4c55-b324-205cacde847c-kube-api-access-vqjlg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:02.974519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:02.974481 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:03.048514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.048473 4556 generic.go:356] "Generic (PLEG): container finished" podID="76188f96-56cb-4c55-b324-205cacde847c" containerID="79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8" exitCode=0 Oct 31 03:18:03.048669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.048633 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" event={"ID":"76188f96-56cb-4c55-b324-205cacde847c","Type":"ContainerDied","Data":"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8"} Oct 31 03:18:03.048716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.048673 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" event={"ID":"76188f96-56cb-4c55-b324-205cacde847c","Type":"ContainerDied","Data":"8b52cef9dce97ac115d2a1b164eee49541fc610929d92a23606b6f43a7a88f19"} Oct 31 03:18:03.048716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.048696 4556 scope.go:122] "RemoveContainer" containerID="79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8" Oct 31 03:18:03.048828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.048817 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f" Oct 31 03:18:03.085311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.085256 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:18:03.091820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.091776 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-56afbd-98b7fcf95-m8f4f"] Oct 31 03:18:03.095700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.095472 4556 scope.go:122] "RemoveContainer" containerID="79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8" Oct 31 03:18:03.095981 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:03.095873 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8\": container with ID starting with 79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8 not found: ID does not exist" containerID="79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8" Oct 31 03:18:03.095981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.095917 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8"} err="failed to get container status \"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8\": rpc error: code = NotFound desc = could not find container \"79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8\": container with ID starting with 79fdf08401877a8a3cf8789c460a852c0c1fc5fb8f15954cfe6d7103cfcbbca8 not found: ID does not exist" Oct 31 03:18:03.798952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.798895 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:03.880062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.880027 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:18:03.880233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:03.880107 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:18:04.262505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:04.262470 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76188f96-56cb-4c55-b324-205cacde847c" path="/var/lib/kubelet/pods/76188f96-56cb-4c55-b324-205cacde847c/volumes" Oct 31 03:18:04.720392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:04.720287 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:05.329934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.329894 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:18:05.331121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331093 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" containerName="revoke-keys" Oct 31 03:18:05.331121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331120 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" containerName="revoke-keys" Oct 31 03:18:05.331121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331140 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0d11af69-591e-4878-9c5b-aa9f198100b4" containerName="revoke-keys" Oct 31 03:18:05.331360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331149 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d11af69-591e-4878-9c5b-aa9f198100b4" containerName="revoke-keys" Oct 31 03:18:05.331360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331165 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:05.331360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331173 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:05.331360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331188 4556 cpu_manager.go:414] "containerMap: removing container" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" Oct 31 03:18:05.331360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331195 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" Oct 31 03:18:05.331617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331391 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:05.331617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331426 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" containerName="revoke-keys" Oct 31 03:18:05.331617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331438 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d11af69-591e-4878-9c5b-aa9f198100b4" containerName="revoke-keys" Oct 31 03:18:05.331617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.331451 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="76188f96-56cb-4c55-b324-205cacde847c" containerName="payload-pre-processing" Oct 31 03:18:05.358157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.357319 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.362436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.362386 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:18:05.380263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.380228 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6d1a8e-serving-cert\"" Oct 31 03:18:05.401956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.401909 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.402174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.402148 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67ps\" (UniqueName: \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.402274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.402251 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.402552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.402529 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.419164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.418635 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:05.505354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.504839 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.505354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.504996 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67ps\" (UniqueName: \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.505354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.505076 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.505354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.505147 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.527451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.508046 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.595838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.595720 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.620524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.620384 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:05.639430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.639194 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67ps\" (UniqueName: \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.709134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.708132 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.751746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.751704 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:05.842951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:05.841035 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:06.095363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:06.095323 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:18:06.095880 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:18:06.095510 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ddac89_e777_4dd8_9d82_3c0e275cd5ae.slice/crio-1e8865747523eef1bde9082594c8af05efc680ef5a299b74312dda4ba2527767 WatchSource:0}: Error finding container 1e8865747523eef1bde9082594c8af05efc680ef5a299b74312dda4ba2527767: Status 404 returned error can't find the container with id 1e8865747523eef1bde9082594c8af05efc680ef5a299b74312dda4ba2527767 Oct 31 03:18:06.543387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:06.543350 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:06.544954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:06.544925 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:18:06.682513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:06.682472 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:06.823992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:06.823954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:07.156460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.156406 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:07.157827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.157793 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" event={"ID":"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae","Type":"ContainerStarted","Data":"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280"} Oct 31 03:18:07.158104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.158080 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" event={"ID":"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae","Type":"ContainerStarted","Data":"1e8865747523eef1bde9082594c8af05efc680ef5a299b74312dda4ba2527767"} Oct 31 03:18:07.158252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.158240 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:07.160326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.160300 4556 generic.go:356] "Generic (PLEG): container finished" podID="738565e2-9595-4697-afee-2838686a2645" containerID="fa02f068b53acac7a319cde8ee1ac4308fbb039bf5d53a82bee64cd596639637" exitCode=0 Oct 31 03:18:07.160488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.160336 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerDied","Data":"fa02f068b53acac7a319cde8ee1ac4308fbb039bf5d53a82bee64cd596639637"} Oct 31 03:18:07.160896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.160877 4556 scope.go:122] "RemoveContainer" containerID="fa02f068b53acac7a319cde8ee1ac4308fbb039bf5d53a82bee64cd596639637" Oct 31 03:18:07.163383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.163281 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:07.163732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.163378 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:07.163732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.163450 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:07.237439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.236231 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" podStartSLOduration=1.640363223 podStartE2EDuration="2.236211307s" totalImagesPullingTime="595.848084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:07.233794754 +0000 UTC m=+1365.088811894" watchObservedRunningTime="2026-10-31 03:18:07.236211307 +0000 UTC m=+1365.091228449" Oct 31 03:18:07.753957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.753850 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:07.904917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:07.904881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:08.031649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.031532 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:08.074545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.074500 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:18:08.172429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.172372 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerStarted","Data":"fce3d8367c3bd9f6bed49fd8b1f699b8007e08b44804da3f4231ca40cb629997"} Oct 31 03:18:08.383742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.383640 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:18:08.385036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.385007 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:08.385211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.385193 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:08.385554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.385535 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ca4c51-2abf-4bf1-9706-08d321b33d5a" containerName="istio-proxy" Oct 31 03:18:08.390686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.390661 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.403409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.403376 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fd580-openshift-default-dockercfg-jw65w\"" Oct 31 03:18:08.427581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.427545 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:18:08.458578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.458533 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.530624 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.530699 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.530753 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.530877 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.531036 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.531064 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.531184 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6t7\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.531218 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.534435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.531257 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.639932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.639823 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.640751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.640724 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.641491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.641462 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6t7\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.642081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.642058 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.642742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.642722 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.643280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.643260 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.661318 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.661428 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.661631 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.640649 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.663090 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.663350 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.665163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.643708 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.672436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.643137 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.675040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.675007 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.684454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.682030 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.684454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.683694 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.707433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.706998 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6t7\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\") pod \"e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.727919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.727807 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:08.817316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.817289 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:18:08.821238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.821195 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:08.821513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:08.821481 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:09.155348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.155280 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:09.222013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.221976 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:18:09.222554 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:18:09.222526 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8ee95d_54a2_42bd_a2eb_7e52cce8dd0b.slice/crio-0a44d02d6a47e1169d3d014addbc155c054f49dc0eb29a5e5b7f0eec986e4224 WatchSource:0}: Error finding container 0a44d02d6a47e1169d3d014addbc155c054f49dc0eb29a5e5b7f0eec986e4224: Status 404 returned error can't find the container with id 0a44d02d6a47e1169d3d014addbc155c054f49dc0eb29a5e5b7f0eec986e4224 Oct 31 03:18:09.226256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.226219 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:09.226372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.226337 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:09.226455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.226387 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:18:09.301437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.298217 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:09.443438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.441916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:09.812722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.812541 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:09.812722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:09.812612 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:10.131578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.130072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:10.239470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.239409 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerStarted","Data":"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd"} Oct 31 03:18:10.239755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.239735 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerStarted","Data":"0a44d02d6a47e1169d3d014addbc155c054f49dc0eb29a5e5b7f0eec986e4224"} Oct 31 03:18:10.266161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.266102 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podStartSLOduration=2.266084131 podStartE2EDuration="2.266084131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:10.262051205 +0000 UTC m=+1368.117068347" watchObservedRunningTime="2026-10-31 03:18:10.266084131 +0000 UTC m=+1368.121101273" Oct 31 03:18:10.498442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.496618 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:10.733604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.733561 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:10.742343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.742295 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:10.807165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.807075 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:10.807165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:10.807140 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:11.246964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.246734 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:11.248541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.248515 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:18:11.267263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.267229 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:11.407817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.407756 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:11.540146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.540008 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:11.569558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.569490 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" podUID="f4367dfd-57bb-4a2b-881c-82651b935470" containerName="payload-processing" containerID="cri-o://01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90" gracePeriod=25 Oct 31 03:18:11.813500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.810892 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:11.813500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:11.810955 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:12.036360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.036314 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:12.129926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.129837 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:18:12.217802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.217766 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:12.222250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.222225 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\") pod \"f4367dfd-57bb-4a2b-881c-82651b935470\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " Oct 31 03:18:12.222372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.222306 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\" (UniqueName: \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\") pod \"f4367dfd-57bb-4a2b-881c-82651b935470\" (UID: \"f4367dfd-57bb-4a2b-881c-82651b935470\") " Oct 31 03:18:12.222806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.222775 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume" pod "f4367dfd-57bb-4a2b-881c-82651b935470" (UID: "f4367dfd-57bb-4a2b-881c-82651b935470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:12.223259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.223235 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4367dfd-57bb-4a2b-881c-82651b935470-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.224727 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.224693 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm" pod "f4367dfd-57bb-4a2b-881c-82651b935470" (UID: "f4367dfd-57bb-4a2b-881c-82651b935470"). InnerVolumeSpecName "kube-api-access-mdstm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:12.263356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.262970 4556 generic.go:356] "Generic (PLEG): container finished" podID="f4367dfd-57bb-4a2b-881c-82651b935470" containerID="01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90" exitCode=0 Oct 31 03:18:12.263962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.263943 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" Oct 31 03:18:12.264123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.264099 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" event={"ID":"f4367dfd-57bb-4a2b-881c-82651b935470","Type":"ContainerDied","Data":"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90"} Oct 31 03:18:12.264175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.264162 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7" event={"ID":"f4367dfd-57bb-4a2b-881c-82651b935470","Type":"ContainerDied","Data":"d231a80819869e92e684cdfb0a68910be87de90423f72fd3bc354dd90034e9f6"} Oct 31 03:18:12.264217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.264189 4556 scope.go:122] "RemoveContainer" containerID="01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90" Oct 31 03:18:12.335027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.334921 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdstm\" (UniqueName: \"kubernetes.io/projected/f4367dfd-57bb-4a2b-881c-82651b935470-kube-api-access-mdstm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:12.361409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.361109 4556 scope.go:122] "RemoveContainer" containerID="01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90" Oct 31 03:18:12.361774 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:12.361734 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90\": container with ID starting with 01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90 not found: ID does not exist" containerID="01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90" Oct 31 03:18:12.361845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.361775 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90"} err="failed to get container status \"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90\": rpc error: code = NotFound desc = could not find container \"01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90\": container with ID starting with 01c225ddd37fcb54020270e0db81731c761dc0e4ba48ea7fb980d8ae27466f90 not found: ID does not exist" Oct 31 03:18:12.368206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.368167 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:18:12.392495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.392355 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-780abd-64f7774c7-fxxc7"] Oct 31 03:18:12.605553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.605516 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:12.806953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.806910 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:12.807157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:12.806973 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:13.196972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:13.196940 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:18:13.497432 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:13.497270 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:13.689660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:13.687753 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:13.808513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:13.807654 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:13.808513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:13.807727 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:14.262815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.262776 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4367dfd-57bb-4a2b-881c-82651b935470" path="/var/lib/kubelet/pods/f4367dfd-57bb-4a2b-881c-82651b935470/volumes" Oct 31 03:18:14.264559 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.264528 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:18:14.264820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.264786 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" containerID="cri-o://207a55dc4c745a637fa39903b46e11f2c44c36b2411a8193349ccf63dcfac048" gracePeriod=30 Oct 31 03:18:14.312276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.312212 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:18:14.312578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.312545 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:18:14.312791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.312756 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" containerID="cri-o://f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3" gracePeriod=30 Oct 31 03:18:14.317971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.317933 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" containerID="cri-o://1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e" gracePeriod=30 Oct 31 03:18:14.684116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.684079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:14.806889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.806847 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 31 03:18:14.808766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.806923 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 31 03:18:14.844346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.836005 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:14.970518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:14.970121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:15.159609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:15.157897 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:15.812183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:15.812150 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:18:15.813484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:15.812645 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:18:15.813484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:15.813457 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:18:15.941243 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:15.941206 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:18:16.429656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:16.429584 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:17.411436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:17.405189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:18.265252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:18.265217 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:18.392767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:18.392729 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:19.146874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.146816 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:19.393208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.392890 4556 generic.go:356] "Generic (PLEG): container finished" podID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerID="207a55dc4c745a637fa39903b46e11f2c44c36b2411a8193349ccf63dcfac048" exitCode=0 Oct 31 03:18:19.393208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.393148 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" event={"ID":"ddf83620-6ddc-4c31-8c8c-e819239caca2","Type":"ContainerDied","Data":"207a55dc4c745a637fa39903b46e11f2c44c36b2411a8193349ccf63dcfac048"} Oct 31 03:18:19.645664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.645634 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:18:19.724126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.724092 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:18:19.766501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.766472 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775541 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775610 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775644 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775749 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775780 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775809 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775840 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775914 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.775994 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776022 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776075 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776105 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776219 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776341 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776401 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776450 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776515 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\") pod \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\" (UID: \"2a9d0cb4-dddd-44bb-be58-a687e98c307d\") " Oct 31 03:18:19.777118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.776568 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\") pod \"ddf83620-6ddc-4c31-8c8c-e819239caca2\" (UID: \"ddf83620-6ddc-4c31-8c8c-e819239caca2\") " Oct 31 03:18:19.780477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.780183 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.780977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.780633 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.781534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.781508 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:19.782309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.782279 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.783027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.782522 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.783027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.782608 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.783555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.783471 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:19.784249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.783784 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.784249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.784016 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.784249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.784090 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.784577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.784549 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.784691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.784670 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "kube-api-access-fv7lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.785488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.785461 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:19.786284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.786258 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.786837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.786810 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:19.788177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.788149 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "kube-api-access-9jdbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.841549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.841475 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy" pod "ddf83620-6ddc-4c31-8c8c-e819239caca2" (UID: "ddf83620-6ddc-4c31-8c8c-e819239caca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.843348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.843306 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token" pod "2a9d0cb4-dddd-44bb-be58-a687e98c307d" (UID: "2a9d0cb4-dddd-44bb-be58-a687e98c307d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.878347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878313 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.878598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878530 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.878671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878649 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.878746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878730 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.878832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878820 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.878894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878864 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.878965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878950 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.879025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.878996 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.879081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879060 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.879139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879089 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.879139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879099 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\") pod \"01a34a51-f8e3-4da6-98d5-6638b944102c\" (UID: \"01a34a51-f8e3-4da6-98d5-6638b944102c\") " Oct 31 03:18:19.879139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879117 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.879385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879354 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.879545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.879487 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:19.880125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880100 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880131 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdbh\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-kube-api-access-9jdbh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880150 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddf83620-6ddc-4c31-8c8c-e819239caca2-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880163 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880179 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880193 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880207 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880221 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880236 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880252 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880266 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880281 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv7lz\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-kube-api-access-fv7lz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880296 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880311 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880324 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880339 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880353 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a9d0cb4-dddd-44bb-be58-a687e98c307d-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880367 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddf83620-6ddc-4c31-8c8c-e819239caca2-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880383 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880407 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddf83620-6ddc-4c31-8c8c-e819239caca2-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880438 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880452 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a34a51-f8e3-4da6-98d5-6638b944102c-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.880728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.880467 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a9d0cb4-dddd-44bb-be58-a687e98c307d-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.881725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.881323 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.881725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.881592 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "kube-api-access-bpv7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:19.881834 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.881760 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:19.882484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.882459 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy" pod "01a34a51-f8e3-4da6-98d5-6638b944102c" (UID: "01a34a51-f8e3-4da6-98d5-6638b944102c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:19.975631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.975526 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:19.982820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.982779 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.982820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.982817 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.983101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.982829 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.983101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.982845 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpv7n\" (UniqueName: \"kubernetes.io/projected/01a34a51-f8e3-4da6-98d5-6638b944102c-kube-api-access-bpv7n\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:19.997551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:19.997470 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:18:20.429498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.429399 4556 generic.go:356] "Generic (PLEG): container finished" podID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerID="1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e" exitCode=0 Oct 31 03:18:20.429965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.429595 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" event={"ID":"01a34a51-f8e3-4da6-98d5-6638b944102c","Type":"ContainerDied","Data":"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e"} Oct 31 03:18:20.429965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.429681 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" event={"ID":"01a34a51-f8e3-4da6-98d5-6638b944102c","Type":"ContainerDied","Data":"cf3e982d12071cca5b05a21c2bc572727220d040fbb1cce574e06377ad816956"} Oct 31 03:18:20.429965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.429710 4556 scope.go:122] "RemoveContainer" containerID="1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e" Oct 31 03:18:20.429965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.429869 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq" Oct 31 03:18:20.435738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.435690 4556 generic.go:356] "Generic (PLEG): container finished" podID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerID="f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3" exitCode=0 Oct 31 03:18:20.435985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.435776 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" Oct 31 03:18:20.437017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.435796 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" event={"ID":"2a9d0cb4-dddd-44bb-be58-a687e98c307d","Type":"ContainerDied","Data":"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3"} Oct 31 03:18:20.437147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.437034 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m" event={"ID":"2a9d0cb4-dddd-44bb-be58-a687e98c307d","Type":"ContainerDied","Data":"c75cac00f252adf2e83cdba31cf3453acee67dfa25c4b2ace517f5e479bff5a4"} Oct 31 03:18:20.438598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.438574 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" Oct 31 03:18:20.438952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.438814 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj" event={"ID":"ddf83620-6ddc-4c31-8c8c-e819239caca2","Type":"ContainerDied","Data":"071602c8de5a13884912aad56d1f4de1a9b2856a5dff3c07c8d40b2b559d0274"} Oct 31 03:18:20.448566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.448537 4556 scope.go:122] "RemoveContainer" containerID="1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e" Oct 31 03:18:20.449734 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:20.449707 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e\": container with ID starting with 1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e not found: ID does not exist" containerID="1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e" Oct 31 03:18:20.449906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.449793 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e"} err="failed to get container status \"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e\": rpc error: code = NotFound desc = could not find container \"1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e\": container with ID starting with 1ba2017b3fefde7e8ddeb4ba5bcfdca77efffb7c543f8b2034e6b1fb7e994a1e not found: ID does not exist" Oct 31 03:18:20.449906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.449821 4556 scope.go:122] "RemoveContainer" containerID="f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3" Oct 31 03:18:20.459190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.459051 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:18:20.466149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.466125 4556 scope.go:122] "RemoveContainer" containerID="f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3" Oct 31 03:18:20.466511 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:20.466482 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3\": container with ID starting with f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3 not found: ID does not exist" containerID="f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3" Oct 31 03:18:20.466612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.466522 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3"} err="failed to get container status \"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3\": rpc error: code = NotFound desc = could not find container \"f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3\": container with ID starting with f138d1fe1fc51bc51d5f82f80c8f1f6ecafb1bb1229e23244243d13c267126d3 not found: ID does not exist" Oct 31 03:18:20.466612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.466545 4556 scope.go:122] "RemoveContainer" containerID="207a55dc4c745a637fa39903b46e11f2c44c36b2411a8193349ccf63dcfac048" Oct 31 03:18:20.467715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.467690 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9bhdq"] Oct 31 03:18:20.492515 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.492358 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:18:20.497898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.497861 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-68xdj"] Oct 31 03:18:20.512592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.512557 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:18:20.517272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.517235 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:18:20.524274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.524243 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-9279m"] Oct 31 03:18:20.533134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.531286 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:18:20.658953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.657553 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:20.812438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:20.807945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:21.061789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:21.061674 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:21.807284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:21.807246 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:21.950529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:21.950403 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:22.179229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.170129 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:18:22.179229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.170380 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" containerID="cri-o://c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8" gracePeriod=30 Oct 31 03:18:22.221440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.197354 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:22.433612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.411143 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:18:22.537636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.527814 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" containerID="cri-o://61071aa4f42af301f776e1783c2fe3589d9007e395d1eb48644fb021f3e3bfe2" gracePeriod=30 Oct 31 03:18:22.877502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.862380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:22.889563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.889518 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:22.948479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.948441 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" path="/var/lib/kubelet/pods/01a34a51-f8e3-4da6-98d5-6638b944102c/volumes" Oct 31 03:18:22.952079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.952044 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" path="/var/lib/kubelet/pods/2a9d0cb4-dddd-44bb-be58-a687e98c307d/volumes" Oct 31 03:18:22.954392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.954327 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" path="/var/lib/kubelet/pods/ddf83620-6ddc-4c31-8c8c-e819239caca2/volumes" Oct 31 03:18:22.999465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:22.997658 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:18:23.051456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:23.043214 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:23.606006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:23.605952 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:18:24.062436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:24.047685 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:25.260435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.260175 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:25.518806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.518687 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" podUID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" containerName="payload-processing" containerID="cri-o://9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9" gracePeriod=25 Oct 31 03:18:25.532374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.532326 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" podUID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" containerName="payload-pre-processing" containerID="cri-o://ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64" gracePeriod=25 Oct 31 03:18:25.882382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.882346 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:18:25.882602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.882579 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" podUID="630cdffc-badd-42d2-9f09-a0babf994da4" containerName="maas-api" containerID="cri-o://9ca37999fbe287e61363de32586a975280299724d972db121c5c5fe9251954c7" gracePeriod=30 Oct 31 03:18:25.886612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.886399 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:18:25.957132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.957099 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:25.961170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:25.961144 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:18:26.000139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.000094 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\" (UniqueName: \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\") pod \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " Oct 31 03:18:26.000510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.000395 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\") pod \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\" (UID: \"582f8ae2-93b2-4a89-8cf9-cf6d922800c8\") " Oct 31 03:18:26.001896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.001858 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume" pod "582f8ae2-93b2-4a89-8cf9-cf6d922800c8" (UID: "582f8ae2-93b2-4a89-8cf9-cf6d922800c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:26.004793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.004756 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg" pod "582f8ae2-93b2-4a89-8cf9-cf6d922800c8" (UID: "582f8ae2-93b2-4a89-8cf9-cf6d922800c8"). InnerVolumeSpecName "kube-api-access-fq8mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.103119 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\") pod \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.103209 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\" (UniqueName: \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\") pod \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\" (UID: \"b68b57e0-574c-43b7-8b6c-40fa82c1d41a\") " Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.103648 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume" pod "b68b57e0-574c-43b7-8b6c-40fa82c1d41a" (UID: "b68b57e0-574c-43b7-8b6c-40fa82c1d41a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.104215 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8mg\" (UniqueName: \"kubernetes.io/projected/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-kube-api-access-fq8mg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.104243 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.104524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.104263 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/582f8ae2-93b2-4a89-8cf9-cf6d922800c8-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.142142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.142095 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972" pod "b68b57e0-574c-43b7-8b6c-40fa82c1d41a" (UID: "b68b57e0-574c-43b7-8b6c-40fa82c1d41a"). InnerVolumeSpecName "kube-api-access-84972". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:26.387742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.248979 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:26.393079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.392737 4556 generic.go:356] "Generic (PLEG): container finished" podID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" containerID="9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9" exitCode=0 Oct 31 03:18:26.393079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.392903 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" event={"ID":"582f8ae2-93b2-4a89-8cf9-cf6d922800c8","Type":"ContainerDied","Data":"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9"} Oct 31 03:18:26.393079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.392958 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" event={"ID":"582f8ae2-93b2-4a89-8cf9-cf6d922800c8","Type":"ContainerDied","Data":"5e38890680c537699ed2d6b77f940ac781f3f4cbcee92aa16eec0b1b8d9db716"} Oct 31 03:18:26.393079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.392991 4556 scope.go:122] "RemoveContainer" containerID="9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9" Oct 31 03:18:26.393079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.393052 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg" Oct 31 03:18:26.396864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.396829 4556 generic.go:356] "Generic (PLEG): container finished" podID="630cdffc-badd-42d2-9f09-a0babf994da4" containerID="9ca37999fbe287e61363de32586a975280299724d972db121c5c5fe9251954c7" exitCode=0 Oct 31 03:18:26.397047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.397027 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" event={"ID":"630cdffc-badd-42d2-9f09-a0babf994da4","Type":"ContainerDied","Data":"9ca37999fbe287e61363de32586a975280299724d972db121c5c5fe9251954c7"} Oct 31 03:18:26.399299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.399083 4556 generic.go:356] "Generic (PLEG): container finished" podID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" containerID="ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64" exitCode=0 Oct 31 03:18:26.399299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.399154 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" event={"ID":"b68b57e0-574c-43b7-8b6c-40fa82c1d41a","Type":"ContainerDied","Data":"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64"} Oct 31 03:18:26.399299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.399210 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" event={"ID":"b68b57e0-574c-43b7-8b6c-40fa82c1d41a","Type":"ContainerDied","Data":"318147da9a01fb80cc98100c3f997218b83b005d05a392dd7ab1f803c2fbd6d4"} Oct 31 03:18:26.399299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.399272 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff" Oct 31 03:18:26.431370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.431336 4556 scope.go:122] "RemoveContainer" containerID="9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9" Oct 31 03:18:26.433859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.433828 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:18:26.436129 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:26.436099 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9\": container with ID starting with 9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9 not found: ID does not exist" containerID="9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9" Oct 31 03:18:26.436302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.436276 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9"} err="failed to get container status \"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9\": rpc error: code = NotFound desc = could not find container \"9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9\": container with ID starting with 9cfcbb89cf413295cf71d7d6f8cd13aabd8864cc3b85b30a4db480f167e788f9 not found: ID does not exist" Oct 31 03:18:26.436434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.436406 4556 scope.go:122] "RemoveContainer" containerID="ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64" Oct 31 03:18:26.442205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.442164 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f1bfbe55-574b595dc-2l6gg"] Oct 31 03:18:26.450820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.450792 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:18:26.461190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.461156 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:18:26.462766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.461983 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84972\" (UniqueName: \"kubernetes.io/projected/b68b57e0-574c-43b7-8b6c-40fa82c1d41a-kube-api-access-84972\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.464130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.463990 4556 scope.go:122] "RemoveContainer" containerID="ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64" Oct 31 03:18:26.464736 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:26.464678 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64\": container with ID starting with ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64 not found: ID does not exist" containerID="ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64" Oct 31 03:18:26.464736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.464708 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64"} err="failed to get container status \"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64\": rpc error: code = NotFound desc = could not find container \"ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64\": container with ID starting with ae547751b2d2da512df075f214015bc2f600ce1f5e42b33c57637aaa85e4ca64 not found: ID does not exist" Oct 31 03:18:26.466729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.466692 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f1bfbe55-d69f8bb79-7f6ff"] Oct 31 03:18:26.518927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.518877 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87"] Oct 31 03:18:26.525373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.525321 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-780abd-75r87"] Oct 31 03:18:26.528321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.528288 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:26.562979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.562934 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\" (UniqueName: \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\") pod \"630cdffc-badd-42d2-9f09-a0babf994da4\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " Oct 31 03:18:26.563158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.563052 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\") pod \"630cdffc-badd-42d2-9f09-a0babf994da4\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " Oct 31 03:18:26.563158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.563145 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") pod \"630cdffc-badd-42d2-9f09-a0babf994da4\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " Oct 31 03:18:26.563502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.563480 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\") pod \"630cdffc-badd-42d2-9f09-a0babf994da4\" (UID: \"630cdffc-badd-42d2-9f09-a0babf994da4\") " Oct 31 03:18:26.563969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.563938 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca" pod "630cdffc-badd-42d2-9f09-a0babf994da4" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:26.564388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.564362 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/630cdffc-badd-42d2-9f09-a0babf994da4-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.565605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.565572 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls" pod "630cdffc-badd-42d2-9f09-a0babf994da4" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:26.565864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.565846 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb" pod "630cdffc-badd-42d2-9f09-a0babf994da4" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4"). InnerVolumeSpecName "kube-api-access-798jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:26.565922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.565850 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs" pod "630cdffc-badd-42d2-9f09-a0babf994da4" (UID: "630cdffc-badd-42d2-9f09-a0babf994da4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:26.656183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.656124 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:26.666245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.666209 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798jb\" (UniqueName: \"kubernetes.io/projected/630cdffc-badd-42d2-9f09-a0babf994da4-kube-api-access-798jb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.666245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.666244 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:26.666245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:26.666256 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/630cdffc-badd-42d2-9f09-a0babf994da4-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.114058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.113937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:27.337667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.337619 4556 scope.go:122] "RemoveContainer" containerID="7e0de6790d23b116e65af942abd0ef5f5a727385bfbfd4c6792e6ab4e1dc2a82" Oct 31 03:18:27.351351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.351327 4556 scope.go:122] "RemoveContainer" containerID="fde7a463c69afcf651b65ddfc1fa461628fc094779285932b4aa8178658ca2cf" Oct 31 03:18:27.366219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.366193 4556 scope.go:122] "RemoveContainer" containerID="6bb0569d56638a76cdc7d3aaf1318cfcef4c369550dd7fc6b4ab367a47ed8ffe" Oct 31 03:18:27.379627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.379604 4556 scope.go:122] "RemoveContainer" containerID="6396a49d4f135a6747b0fa09e3287c6e493e56fa0aba44922fc8fc4271852cfa" Oct 31 03:18:27.392278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.392248 4556 scope.go:122] "RemoveContainer" containerID="58b6402176affe264ef95dea22a48a15ccc9e3930a03836fc694ab1f414969dc" Oct 31 03:18:27.403698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.403664 4556 scope.go:122] "RemoveContainer" containerID="19a011f09ccb392d4517f9be31190c108ff3faf0d91f1c06c4d9006be9c8cbee" Oct 31 03:18:27.411951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.411911 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" Oct 31 03:18:27.411951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.411920 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r" event={"ID":"630cdffc-badd-42d2-9f09-a0babf994da4","Type":"ContainerDied","Data":"65a84958bf0559ff701136843116c4257bd7815757c4b4e72fb516a4fbf1630f"} Oct 31 03:18:27.412177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.412002 4556 scope.go:122] "RemoveContainer" containerID="9ca37999fbe287e61363de32586a975280299724d972db121c5c5fe9251954c7" Oct 31 03:18:27.438740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.438716 4556 scope.go:122] "RemoveContainer" containerID="cefa458e55338a559db5d1219ba7343351ab3a32cce74f2a932e712135fe1b84" Oct 31 03:18:27.452546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.452508 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:18:27.453584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.453562 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-780abd-5457b76d4d-ggf5r"] Oct 31 03:18:27.515371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.515346 4556 scope.go:122] "RemoveContainer" containerID="9f5c76c265aec2a39cc8eab0038d1a9cc0c9db14f2b95c868d1b3bc59f9590f3" Oct 31 03:18:27.526870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.526845 4556 scope.go:122] "RemoveContainer" containerID="9b932d92cf8c04bb6edda261388fade156aaaf607ef8398197fee3443fd423ed" Oct 31 03:18:27.540805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.540768 4556 scope.go:122] "RemoveContainer" containerID="f30acbeda4ec4fe49d7c8e5047dba912818f478692781ac75ea4039be506e013" Oct 31 03:18:27.725237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.725209 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:18:27.843244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.843201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.889400 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890066 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890154 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890201 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890333 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890373 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890853 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.890977 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.892261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.891007 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\") pod \"b209d0b1-af15-4b48-b249-397a6b722aef\" (UID: \"b209d0b1-af15-4b48-b249-397a6b722aef\") " Oct 31 03:18:27.895683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.895546 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "kube-api-access-z5njs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.897469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.897378 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:27.905167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.905126 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:27.906551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.906507 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.906551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.906536 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.906955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.906910 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.907064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.906987 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.907654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.907596 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:27.908825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.908802 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo" pod "b209d0b1-af15-4b48-b249-397a6b722aef" (UID: "b209d0b1-af15-4b48-b249-397a6b722aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:27.994160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994125 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994162 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994177 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b209d0b1-af15-4b48-b249-397a6b722aef-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994191 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994204 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994218 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994233 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b209d0b1-af15-4b48-b249-397a6b722aef-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994247 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5njs\" (UniqueName: \"kubernetes.io/projected/b209d0b1-af15-4b48-b249-397a6b722aef-kube-api-access-z5njs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:27.994362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:27.994261 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b209d0b1-af15-4b48-b249-397a6b722aef-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.260352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.260311 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d11af69-591e-4878-9c5b-aa9f198100b4" path="/var/lib/kubelet/pods/0d11af69-591e-4878-9c5b-aa9f198100b4/volumes" Oct 31 03:18:28.260979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.260955 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" path="/var/lib/kubelet/pods/582f8ae2-93b2-4a89-8cf9-cf6d922800c8/volumes" Oct 31 03:18:28.261625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.261601 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630cdffc-badd-42d2-9f09-a0babf994da4" path="/var/lib/kubelet/pods/630cdffc-badd-42d2-9f09-a0babf994da4/volumes" Oct 31 03:18:28.262427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.262394 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" path="/var/lib/kubelet/pods/b68b57e0-574c-43b7-8b6c-40fa82c1d41a/volumes" Oct 31 03:18:28.442060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.442021 4556 generic.go:356] "Generic (PLEG): container finished" podID="b209d0b1-af15-4b48-b249-397a6b722aef" containerID="c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8" exitCode=0 Oct 31 03:18:28.442582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.442238 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" event={"ID":"b209d0b1-af15-4b48-b249-397a6b722aef","Type":"ContainerDied","Data":"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8"} Oct 31 03:18:28.442582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.442295 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" event={"ID":"b209d0b1-af15-4b48-b249-397a6b722aef","Type":"ContainerDied","Data":"e3a61d08eb7db995a9f991c867391a8e9cffbc571c662368e78f42c73bfce88b"} Oct 31 03:18:28.442582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.442348 4556 scope.go:122] "RemoveContainer" containerID="c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8" Oct 31 03:18:28.442582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.442499 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9" Oct 31 03:18:28.450786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.450748 4556 generic.go:356] "Generic (PLEG): container finished" podID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerID="61071aa4f42af301f776e1783c2fe3589d9007e395d1eb48644fb021f3e3bfe2" exitCode=0 Oct 31 03:18:28.451097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.451052 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" event={"ID":"fd7abd28-dcc5-4db3-a851-caf8572900e3","Type":"ContainerDied","Data":"61071aa4f42af301f776e1783c2fe3589d9007e395d1eb48644fb021f3e3bfe2"} Oct 31 03:18:28.465045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.465021 4556 scope.go:122] "RemoveContainer" containerID="c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8" Oct 31 03:18:28.465501 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:28.465466 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8\": container with ID starting with c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8 not found: ID does not exist" containerID="c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8" Oct 31 03:18:28.465702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.465509 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8"} err="failed to get container status \"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8\": rpc error: code = NotFound desc = could not find container \"c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8\": container with ID starting with c5085cb82695eed6e50a32d4154e3f5ef491e6df34f5b73b3d47d0e6e987dbf8 not found: ID does not exist" Oct 31 03:18:28.468643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.468612 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:18:28.474523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.474483 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-tqrw9"] Oct 31 03:18:28.519803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.519726 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:18:28.539089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.538936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:28.707555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707511 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707629 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707658 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707737 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707759 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707849 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707893 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.707987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707955 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.708133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.707997 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\") pod \"fd7abd28-dcc5-4db3-a851-caf8572900e3\" (UID: \"fd7abd28-dcc5-4db3-a851-caf8572900e3\") " Oct 31 03:18:28.708133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.708038 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.708242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.708139 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.708242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.708174 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:28.708370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.708316 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.708370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.708360 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.709109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.709079 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.709250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.709114 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7abd28-dcc5-4db3-a851-caf8572900e3-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.709250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.709132 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.709250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.709147 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.709250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.709163 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.710053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.710025 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:18:28.710159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.710077 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:28.710159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.710088 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:28.710487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.710471 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr" pod "fd7abd28-dcc5-4db3-a851-caf8572900e3" (UID: "fd7abd28-dcc5-4db3-a851-caf8572900e3"). InnerVolumeSpecName "kube-api-access-kslfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:28.810700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.810613 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.810700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.810646 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.810700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.810658 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kslfr\" (UniqueName: \"kubernetes.io/projected/fd7abd28-dcc5-4db3-a851-caf8572900e3-kube-api-access-kslfr\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:28.810700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:28.810668 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7abd28-dcc5-4db3-a851-caf8572900e3-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:29.343390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.343322 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:29.460847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.460818 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" Oct 31 03:18:29.461307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.460846 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn" event={"ID":"fd7abd28-dcc5-4db3-a851-caf8572900e3","Type":"ContainerDied","Data":"994ca3c175b0b67a7e5783e25991bee189fe4e28f1077134046ae77aeb6c245b"} Oct 31 03:18:29.461307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.460918 4556 scope.go:122] "RemoveContainer" containerID="61071aa4f42af301f776e1783c2fe3589d9007e395d1eb48644fb021f3e3bfe2" Oct 31 03:18:29.486868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.486829 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:18:29.490884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:29.490856 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-56afbd-openshift-default-6c56486975-9xctn"] Oct 31 03:18:30.003302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.003261 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:18:30.004487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004464 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" Oct 31 03:18:30.004558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004489 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" Oct 31 03:18:30.004558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004511 4556 cpu_manager.go:414] "containerMap: removing container" podUID="630cdffc-badd-42d2-9f09-a0babf994da4" containerName="maas-api" Oct 31 03:18:30.004558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004520 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630cdffc-badd-42d2-9f09-a0babf994da4" containerName="maas-api" Oct 31 03:18:30.004558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004538 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f4367dfd-57bb-4a2b-881c-82651b935470" containerName="payload-processing" Oct 31 03:18:30.004558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004546 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4367dfd-57bb-4a2b-881c-82651b935470" containerName="payload-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004575 4556 cpu_manager.go:414] "containerMap: removing container" podUID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" containerName="payload-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004583 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" containerName="payload-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004603 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" containerName="payload-pre-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004612 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" containerName="payload-pre-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004628 4556 cpu_manager.go:414] "containerMap: removing container" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004635 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004663 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004671 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004684 4556 cpu_manager.go:414] "containerMap: removing container" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004692 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004708 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004715 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004871 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf83620-6ddc-4c31-8c8c-e819239caca2" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004885 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="630cdffc-badd-42d2-9f09-a0babf994da4" containerName="maas-api" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004897 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4367dfd-57bb-4a2b-881c-82651b935470" containerName="payload-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004910 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="582f8ae2-93b2-4a89-8cf9-cf6d922800c8" containerName="payload-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004921 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68b57e0-574c-43b7-8b6c-40fa82c1d41a" containerName="payload-pre-processing" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004930 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a34a51-f8e3-4da6-98d5-6638b944102c" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004943 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9d0cb4-dddd-44bb-be58-a687e98c307d" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004952 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" containerName="istio-proxy" Oct 31 03:18:30.005611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.004961 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" containerName="istio-proxy" Oct 31 03:18:30.011080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.011048 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.015703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.015433 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1fd580-serving-cert\"" Oct 31 03:18:30.019233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.019200 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:18:30.033153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.032531 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.033153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.032746 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.033153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.032886 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.033153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.032921 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5d9\" (UniqueName: \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.118900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.118834 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:30.134149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.134108 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.134149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.134155 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5d9\" (UniqueName: \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.134368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.134241 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.134368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.134354 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.134948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.134915 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.136901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.136875 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.137029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.136997 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.146661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.146622 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5d9\" (UniqueName: \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\") pod \"maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.250351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.248088 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:30.268386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.268292 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b209d0b1-af15-4b48-b249-397a6b722aef" path="/var/lib/kubelet/pods/b209d0b1-af15-4b48-b249-397a6b722aef/volumes" Oct 31 03:18:30.269916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.269884 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7abd28-dcc5-4db3-a851-caf8572900e3" path="/var/lib/kubelet/pods/fd7abd28-dcc5-4db3-a851-caf8572900e3/volumes" Oct 31 03:18:30.346872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.346829 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:30.721435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.707059 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:18:30.927882 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:30.927846 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:31.130832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.130740 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:31.484136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.484102 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" event={"ID":"48071f9d-c984-450e-b06e-69eef341d749","Type":"ContainerStarted","Data":"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c"} Oct 31 03:18:31.484346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.484164 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" event={"ID":"48071f9d-c984-450e-b06e-69eef341d749","Type":"ContainerStarted","Data":"e69412c7c80b83ec82cc2486f5a79f0351697eb818bd1dcfe3584386e5a90929"} Oct 31 03:18:31.484346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.484194 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:31.502599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.502547 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" podStartSLOduration=2.009451565 podStartE2EDuration="2.502531933s" totalImagesPullingTime="493.080368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:31.501994899 +0000 UTC m=+1389.357012039" watchObservedRunningTime="2026-10-31 03:18:31.502531933 +0000 UTC m=+1389.357549073" Oct 31 03:18:31.945148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:31.944162 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:33.010429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:33.010378 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:33.734237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:33.731279 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:34.152613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:34.152582 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:34.275339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:34.275294 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:34.529280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:34.529181 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:34.722641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:34.722602 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:35.470113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:35.470080 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:35.604977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:35.604941 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:35.970441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:35.959376 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:36.137438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:36.137126 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:36.457615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:36.457585 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:36.637437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:36.637139 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:36.816811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:36.814105 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:37.118562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:37.118381 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:37.251794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:37.251731 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:37.499339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:37.499309 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:18:38.045060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:38.045012 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:38.829856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:38.829801 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:39.834928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:39.834748 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:40.048751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:40.048710 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:40.251710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:40.251643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:40.947240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:40.947189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.240102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.240003 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:18:41.246144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.246106 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.251342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.251191 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a2bec7\"" Oct 31 03:18:41.253398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.253337 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.257365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.257199 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:18:41.329058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.329018 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.329385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.329362 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjj6\" (UniqueName: \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.441482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.427713 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.454660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.445932 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjj6\" (UniqueName: \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.454660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.446037 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.454660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.447356 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.506499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.501337 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjj6\" (UniqueName: \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\") pod \"payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.573807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.573758 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:41.934891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.934842 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.954144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:41.954106 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:18:42.199866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.199746 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:42.351435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.348027 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:42.675169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.675128 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" event={"ID":"cefa6a8e-9cfa-4809-83be-a2ba264b914c","Type":"ContainerStarted","Data":"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a"} Oct 31 03:18:42.675353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.675198 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" event={"ID":"cefa6a8e-9cfa-4809-83be-a2ba264b914c","Type":"ContainerStarted","Data":"8e9a7e70a357800c48891ec76737db2c1e38b95aa4050fa65e1f227d4271a250"} Oct 31 03:18:42.676528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.676501 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:42.700668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.700607 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" podStartSLOduration=1.700590769 podStartE2EDuration="1.700590769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:42.696229556 +0000 UTC m=+1400.551246697" watchObservedRunningTime="2026-10-31 03:18:42.700590769 +0000 UTC m=+1400.555607911" Oct 31 03:18:42.925701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:42.925591 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:43.645743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:43.645688 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:44.470493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:44.470449 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:45.262860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:45.257884 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:46.036066 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:46.030244 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:18:46.290269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:46.290181 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:47.027719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:47.027680 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:47.807188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:47.807148 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:48.038034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:48.037976 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w"] Oct 31 03:18:48.041956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:48.041929 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06eae1b-tbt8w"] Oct 31 03:18:48.260058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:48.260021 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91abd35-c4ed-4285-b927-5d31237b52a1" path="/var/lib/kubelet/pods/a91abd35-c4ed-4285-b927-5d31237b52a1/volumes" Oct 31 03:18:48.644882 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:48.644790 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:48.783599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:48.783562 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:49.445491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:49.445447 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:49.583705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:49.583665 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:49.905440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:49.905372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:50.197555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:50.189399 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:50.635390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:50.635097 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:51.170123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.170081 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:18:51.170470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.170408 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" podUID="0349766b-07a9-47c1-b1ea-66971f8354b7" containerName="maas-api" containerID="cri-o://a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303" gracePeriod=30 Oct 31 03:18:51.250821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.250781 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:51.471955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.469072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:51.616342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.616270 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:51.692913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.692882 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:18:51.751357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.751093 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:51.771484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.771388 4556 generic.go:356] "Generic (PLEG): container finished" podID="0349766b-07a9-47c1-b1ea-66971f8354b7" containerID="a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303" exitCode=0 Oct 31 03:18:51.771671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.771585 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" event={"ID":"0349766b-07a9-47c1-b1ea-66971f8354b7","Type":"ContainerDied","Data":"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303"} Oct 31 03:18:51.771671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.771632 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" event={"ID":"0349766b-07a9-47c1-b1ea-66971f8354b7","Type":"ContainerDied","Data":"968d11e73662752d5ac6d7e847df3119472736adb79d793601cb3619918dbf9f"} Oct 31 03:18:51.771671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.771656 4556 scope.go:122] "RemoveContainer" containerID="a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303" Oct 31 03:18:51.771831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.771818 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k" Oct 31 03:18:51.774365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.774331 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\") pod \"0349766b-07a9-47c1-b1ea-66971f8354b7\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " Oct 31 03:18:51.774544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.774447 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\") pod \"0349766b-07a9-47c1-b1ea-66971f8354b7\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " Oct 31 03:18:51.774544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.774497 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\") pod \"0349766b-07a9-47c1-b1ea-66971f8354b7\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " Oct 31 03:18:51.774632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.774580 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\" (UniqueName: \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\") pod \"0349766b-07a9-47c1-b1ea-66971f8354b7\" (UID: \"0349766b-07a9-47c1-b1ea-66971f8354b7\") " Oct 31 03:18:51.777065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.777035 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca" pod "0349766b-07a9-47c1-b1ea-66971f8354b7" (UID: "0349766b-07a9-47c1-b1ea-66971f8354b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:51.779895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.779844 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk" pod "0349766b-07a9-47c1-b1ea-66971f8354b7" (UID: "0349766b-07a9-47c1-b1ea-66971f8354b7"). InnerVolumeSpecName "kube-api-access-8cmzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:51.780066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.779948 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs" pod "0349766b-07a9-47c1-b1ea-66971f8354b7" (UID: "0349766b-07a9-47c1-b1ea-66971f8354b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:51.780185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.780089 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls" pod "0349766b-07a9-47c1-b1ea-66971f8354b7" (UID: "0349766b-07a9-47c1-b1ea-66971f8354b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:51.816357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.816324 4556 scope.go:122] "RemoveContainer" containerID="a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303" Oct 31 03:18:51.816874 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:18:51.816832 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303\": container with ID starting with a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303 not found: ID does not exist" containerID="a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303" Oct 31 03:18:51.817011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.816886 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303"} err="failed to get container status \"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303\": rpc error: code = NotFound desc = could not find container \"a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303\": container with ID starting with a45a9d37fa815e24f44d54b6774f96eeb1c93b9bfb3c683a1d1e43f289bbc303 not found: ID does not exist" Oct 31 03:18:51.876995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.876960 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:51.876995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.876993 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0349766b-07a9-47c1-b1ea-66971f8354b7-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:51.877189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.877005 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0349766b-07a9-47c1-b1ea-66971f8354b7-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:51.877189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:51.877014 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cmzk\" (UniqueName: \"kubernetes.io/projected/0349766b-07a9-47c1-b1ea-66971f8354b7-kube-api-access-8cmzk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:52.046173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.046079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:52.098316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.098259 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:18:52.102508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.102468 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b93a15-5f45cb8f59-xnm7k"] Oct 31 03:18:52.263246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.263204 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0349766b-07a9-47c1-b1ea-66971f8354b7" path="/var/lib/kubelet/pods/0349766b-07a9-47c1-b1ea-66971f8354b7/volumes" Oct 31 03:18:52.744106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.744063 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:18:52.745242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.745213 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0349766b-07a9-47c1-b1ea-66971f8354b7" containerName="maas-api" Oct 31 03:18:52.745306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.745246 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0349766b-07a9-47c1-b1ea-66971f8354b7" containerName="maas-api" Oct 31 03:18:52.745560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.745541 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0349766b-07a9-47c1-b1ea-66971f8354b7" containerName="maas-api" Oct 31 03:18:52.751525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.751395 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.758223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.758193 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a2bec7-serving-cert\"" Oct 31 03:18:52.777062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.777011 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:18:52.791935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.791895 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzwq\" (UniqueName: \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.792196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.792173 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.792302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.792249 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.792371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.792340 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.824958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.824923 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:52.894989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.894953 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzwq\" (UniqueName: \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.895173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.895129 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.895239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.895184 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.895239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.895211 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.895857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.895830 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.897843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.897818 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.897843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.897836 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.908340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.908303 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzwq\" (UniqueName: \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\") pod \"maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:52.973436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:52.969189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:53.085202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:53.085108 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:53.397307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:53.396997 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:18:53.400651 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:18:53.400616 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f103c4_8022_4f07_a6be_b76dfa9f1ccc.slice/crio-7d894642520ff64ea71eeba9c6e9f7d97fad996543cf4683a9920b3604375fdd WatchSource:0}: Error finding container 7d894642520ff64ea71eeba9c6e9f7d97fad996543cf4683a9920b3604375fdd: Status 404 returned error can't find the container with id 7d894642520ff64ea71eeba9c6e9f7d97fad996543cf4683a9920b3604375fdd Oct 31 03:18:53.802485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:53.802402 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" event={"ID":"07f103c4-8022-4f07-a6be-b76dfa9f1ccc","Type":"ContainerStarted","Data":"7d894642520ff64ea71eeba9c6e9f7d97fad996543cf4683a9920b3604375fdd"} Oct 31 03:18:54.711450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:54.711398 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:18:54.813030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:54.812998 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" event={"ID":"07f103c4-8022-4f07-a6be-b76dfa9f1ccc","Type":"ContainerStarted","Data":"5c7f13eb95ccc01826b02028670751a456d97926d8a49eb829fa4eb3c06cf867"} Oct 31 03:18:54.813490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:54.813199 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:18:54.839267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:54.839211 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" podStartSLOduration=2.440369655 podStartE2EDuration="2.839193179s" totalImagesPullingTime="398.823524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:54.836321217 +0000 UTC m=+1412.691338360" watchObservedRunningTime="2026-10-31 03:18:54.839193179 +0000 UTC m=+1412.694210370" Oct 31 03:18:55.939234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.939189 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:18:55.944400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.944374 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:55.948654 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.948625 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3b8712-dockercfg-vfvg5\"" Oct 31 03:18:55.953510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.953484 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3b8712\"" Oct 31 03:18:55.953993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.953966 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:18:55.978024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:55.977981 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:56.075779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.075734 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkx7k\" (UniqueName: \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.076087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.075846 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.103736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.103694 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:56.178712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.178666 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkx7k\" (UniqueName: \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.178936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.178776 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.179872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.179837 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.204270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.204189 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkx7k\" (UniqueName: \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\") pod \"payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.273463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.272188 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.318115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.318040 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:56.444233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.444188 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:56.622579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.622545 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:18:56.622979 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:18:56.622944 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6446de04_d3f8_4a36_ae3d_35a23618359a.slice/crio-00bd6b3700739846e5719e73569945149ee09e5f6a614d06c73c3c1e946e7b30 WatchSource:0}: Error finding container 00bd6b3700739846e5719e73569945149ee09e5f6a614d06c73c3c1e946e7b30: Status 404 returned error can't find the container with id 00bd6b3700739846e5719e73569945149ee09e5f6a614d06c73c3c1e946e7b30 Oct 31 03:18:56.829557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.829520 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" event={"ID":"6446de04-d3f8-4a36-ae3d-35a23618359a","Type":"ContainerStarted","Data":"8f675182544cce70118f92f29a2a177b29d80654efe2345a6bbb90a9220708c6"} Oct 31 03:18:56.829716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.829578 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" event={"ID":"6446de04-d3f8-4a36-ae3d-35a23618359a","Type":"ContainerStarted","Data":"00bd6b3700739846e5719e73569945149ee09e5f6a614d06c73c3c1e946e7b30"} Oct 31 03:18:56.830641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.830620 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:18:56.852082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.851998 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" podStartSLOduration=1.851974622 podStartE2EDuration="1.851974622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:56.846497931 +0000 UTC m=+1414.701515083" watchObservedRunningTime="2026-10-31 03:18:56.851974622 +0000 UTC m=+1414.706991764" Oct 31 03:18:56.916973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:56.916936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:57.913052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:57.912999 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:58.728716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:58.728621 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:59.279575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:59.279523 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:18:59.844437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:18:59.829764 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:00.422433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.420073 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:00.551436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.550904 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:00.747053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.746819 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:00.846520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.834248 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:19:00.847211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.847166 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" containerID="cri-o://fce3d8367c3bd9f6bed49fd8b1f699b8007e08b44804da3f4231ca40cb629997" gracePeriod=30 Oct 31 03:19:00.922446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.909512 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:19:00.935195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.922698 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 03:19:00.935195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.922818 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:19:00.935195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:00.923887 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:01.955295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:01.955251 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:02.938500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:02.938458 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:03.730442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:03.730375 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:04.451445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:04.448052 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:05.156346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:05.153994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:05.855123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:05.855041 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:06.107502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.106742 4556 generic.go:356] "Generic (PLEG): container finished" podID="738565e2-9595-4697-afee-2838686a2645" containerID="fce3d8367c3bd9f6bed49fd8b1f699b8007e08b44804da3f4231ca40cb629997" exitCode=0 Oct 31 03:19:06.107502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.106815 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerDied","Data":"fce3d8367c3bd9f6bed49fd8b1f699b8007e08b44804da3f4231ca40cb629997"} Oct 31 03:19:06.107502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.106914 4556 scope.go:122] "RemoveContainer" containerID="fa02f068b53acac7a319cde8ee1ac4308fbb039bf5d53a82bee64cd596639637" Oct 31 03:19:06.618085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.617645 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:06.719634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.719542 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:19:06.825138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.824813 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.825138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.824899 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.825138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.824935 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.825138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825010 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825317 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825317 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825490 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825526 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.825587 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\") pod \"738565e2-9595-4697-afee-2838686a2645\" (UID: \"738565e2-9595-4697-afee-2838686a2645\") " Oct 31 03:19:06.826567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.826541 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.826828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.826802 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:06.827652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.827622 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:06.827794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.827704 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:06.829265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.827954 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:06.838621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.838576 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:06.840188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.840151 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:06.840803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.840769 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "kube-api-access-t462f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:06.843915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.843874 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token" pod "738565e2-9595-4697-afee-2838686a2645" (UID: "738565e2-9595-4697-afee-2838686a2645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:06.927754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927711 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/738565e2-9595-4697-afee-2838686a2645-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.927754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927748 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.927754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927759 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.928008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927767 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/738565e2-9595-4697-afee-2838686a2645-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.928008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927777 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t462f\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-kube-api-access-t462f\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.928008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927787 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/738565e2-9595-4697-afee-2838686a2645-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.928008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927799 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:06.928008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:06.927807 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/738565e2-9595-4697-afee-2838686a2645-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:07.116725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.116638 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" Oct 31 03:19:07.116725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.116673 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z" event={"ID":"738565e2-9595-4697-afee-2838686a2645","Type":"ContainerDied","Data":"ebe9e16c624936553818b8800d216f6dded12e4eabdc2db9a93bf5c1733e0c92"} Oct 31 03:19:07.116958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.116748 4556 scope.go:122] "RemoveContainer" containerID="fce3d8367c3bd9f6bed49fd8b1f699b8007e08b44804da3f4231ca40cb629997" Oct 31 03:19:07.142307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.142257 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:19:07.146604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.146570 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b93a15-openshift-default-94fdcc9cc-85x9z"] Oct 31 03:19:07.311918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.311860 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:07.640680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:07.640644 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:08.292635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.292598 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738565e2-9595-4697-afee-2838686a2645" path="/var/lib/kubelet/pods/738565e2-9595-4697-afee-2838686a2645/volumes" Oct 31 03:19:08.293148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.293123 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:19:08.294201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.294172 4556 cpu_manager.go:414] "containerMap: removing container" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.294201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.294202 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.298116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.298092 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.309746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.309699 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:19:08.309945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.309836 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.313434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.313388 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-62bf00-openshift-default-dockercfg-xf6pr\"" Oct 31 03:19:08.467228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.464243 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.473678 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.473770 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.473815 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.473845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9qg\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.473960 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.474032 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.474081 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.474108 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.537554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.474183 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.541452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.538557 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:19:08.541452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.539911 4556 cpu_manager.go:414] "containerMap: removing container" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.541452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.539933 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.541452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.540087 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="738565e2-9595-4697-afee-2838686a2645" containerName="istio-proxy" Oct 31 03:19:08.566489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.563877 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:19:08.566489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.563996 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579273 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579373 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579442 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579466 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579538 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579763 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579836 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579879 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.579909 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9qg\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.581229 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.581526 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.581797 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.581912 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.582441 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.596369 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.600661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.600287 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9qg\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.607051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.601695 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.607051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.603425 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-44dd9\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.617945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.617897 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3b8712-serving-cert\"" Oct 31 03:19:08.672142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.671059 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:08.682612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.682550 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.682800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.682630 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.682800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.682721 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgmq\" (UniqueName: \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.682932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.682869 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.683487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.683464 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:08.785474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.785380 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.785774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.785477 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.785774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.785555 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgmq\" (UniqueName: \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.785774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.785690 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.786244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.786201 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.789993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.789292 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.790843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.790752 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.798725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.798680 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgmq\" (UniqueName: \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\") pod \"maas-api-e2e-api-b-3b8712-69954d7445-9b7kz\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.805392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.805351 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:08.868601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.864819 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:19:08.926514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.924869 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:08.954514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:08.950658 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:09.076823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.076771 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:19:09.085160 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:09.085078 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137b2f24_7930_414e_acec_df30d6a1f713.slice/crio-70fd71955082392d41d1aef4d3879e03da468114ce9e1cb8ac9bc7bab840b38e WatchSource:0}: Error finding container 70fd71955082392d41d1aef4d3879e03da468114ce9e1cb8ac9bc7bab840b38e: Status 404 returned error can't find the container with id 70fd71955082392d41d1aef4d3879e03da468114ce9e1cb8ac9bc7bab840b38e Oct 31 03:19:09.091475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.089007 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:09.091475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.089088 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:09.091475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.089117 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:09.154068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.153897 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerStarted","Data":"70fd71955082392d41d1aef4d3879e03da468114ce9e1cb8ac9bc7bab840b38e"} Oct 31 03:19:09.215923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.214128 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:09.303436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.303375 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:19:09.316020 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:09.315979 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903a5bf_4b80_4f2a_8736_f0bd1a40792e.slice/crio-1094fb1a66a6e9ef7a90a642f57d7bc5462d8dcbb7334be34ee1c46f64b20046 WatchSource:0}: Error finding container 1094fb1a66a6e9ef7a90a642f57d7bc5462d8dcbb7334be34ee1c46f64b20046: Status 404 returned error can't find the container with id 1094fb1a66a6e9ef7a90a642f57d7bc5462d8dcbb7334be34ee1c46f64b20046 Oct 31 03:19:09.958939 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:09.958701 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:10.165710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.165668 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerStarted","Data":"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8"} Oct 31 03:19:10.168101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.168055 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" event={"ID":"e903a5bf-4b80-4f2a-8736-f0bd1a40792e","Type":"ContainerStarted","Data":"d671ec234da92ce5872cf3087e4e253f228bc12d7672d3df1e703f9ceea04180"} Oct 31 03:19:10.168282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.168122 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" event={"ID":"e903a5bf-4b80-4f2a-8736-f0bd1a40792e","Type":"ContainerStarted","Data":"1094fb1a66a6e9ef7a90a642f57d7bc5462d8dcbb7334be34ee1c46f64b20046"} Oct 31 03:19:10.168282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.168256 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:10.191163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.191075 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podStartSLOduration=2.191055974 podStartE2EDuration="2.191055974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:10.190554197 +0000 UTC m=+1428.045571342" watchObservedRunningTime="2026-10-31 03:19:10.191055974 +0000 UTC m=+1428.046073119" Oct 31 03:19:10.218741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.218621 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" podStartSLOduration=1.570245237 podStartE2EDuration="2.218599227s" totalImagesPullingTime="648.35399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:10.210140682 +0000 UTC m=+1428.065157826" watchObservedRunningTime="2026-10-31 03:19:10.218599227 +0000 UTC m=+1428.073616368" Oct 31 03:19:10.440542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.440505 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:10.692335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.692281 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:10.699008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.698972 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:10.806127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.806090 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:10.938500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:10.938466 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:11.179208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.179147 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:11.180271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.180233 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:19:11.409439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.402760 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:19:11.409439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.403006 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" podUID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" containerName="maas-api" containerID="cri-o://dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280" gracePeriod=30 Oct 31 03:19:11.494379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.457068 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:11.618081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.617938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:11.769564 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.769388 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:11.984801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.967831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:11.984801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.967913 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:19:12.030455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:11.990292 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:19:12.115840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.112880 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:12.197482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.187157 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:19:12.199927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.199891 4556 generic.go:356] "Generic (PLEG): container finished" podID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" containerID="dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280" exitCode=0 Oct 31 03:19:12.200395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.200145 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" event={"ID":"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae","Type":"ContainerDied","Data":"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280"} Oct 31 03:19:12.200395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.200223 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" event={"ID":"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae","Type":"ContainerDied","Data":"1e8865747523eef1bde9082594c8af05efc680ef5a299b74312dda4ba2527767"} Oct 31 03:19:12.200395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.200257 4556 scope.go:122] "RemoveContainer" containerID="dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280" Oct 31 03:19:12.211478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.206902 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\") pod \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " Oct 31 03:19:12.211478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.206969 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\") pod \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " Oct 31 03:19:12.211478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.207061 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\") pod \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " Oct 31 03:19:12.211478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.207197 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\" (UniqueName: \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\") pod \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\" (UID: \"f6ddac89-e777-4dd8-9d82-3c0e275cd5ae\") " Oct 31 03:19:12.219844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.216885 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca" pod "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" (UID: "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:12.235730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.235580 4556 scope.go:122] "RemoveContainer" containerID="dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280" Oct 31 03:19:12.236318 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:12.236232 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280\": container with ID starting with dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280 not found: ID does not exist" containerID="dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280" Oct 31 03:19:12.236318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.236271 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280"} err="failed to get container status \"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280\": rpc error: code = NotFound desc = could not find container \"dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280\": container with ID starting with dfc8f56f9f2d5bdc2d115bb1265091b70a9203da657c1f680e2f771f6b485280 not found: ID does not exist" Oct 31 03:19:12.255788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.249652 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls" pod "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" (UID: "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:12.255788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.249737 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs" pod "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" (UID: "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:12.331455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.324692 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:12.332493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.332468 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:12.338680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.338643 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:12.406088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.404015 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:19:12.406088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.404090 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:19:12.406088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.404648 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps" pod "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" (UID: "f6ddac89-e777-4dd8-9d82-3c0e275cd5ae"). InnerVolumeSpecName "kube-api-access-r67ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:12.499549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.499504 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67ps\" (UniqueName: \"kubernetes.io/projected/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae-kube-api-access-r67ps\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:12.695386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.695346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:12.819845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.819781 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:12.989755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:12.989655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.119954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.208949 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6" Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.253088 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.257654 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.259258 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6d1a8e-647bc6b87d-qtrc6"] Oct 31 03:19:14.085225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:13.786313 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.188053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.188011 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:19:14.188855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.188827 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.189078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.189058 4556 cpu_manager.go:414] "containerMap: removing container" podUID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" containerName="maas-api" Oct 31 03:19:14.189078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.189077 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" containerName="maas-api" Oct 31 03:19:14.189270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.189203 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" containerName="maas-api" Oct 31 03:19:14.228945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.228902 4556 generic.go:356] "Generic (PLEG): container finished" podID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerID="c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd" exitCode=0 Oct 31 03:19:14.249061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.249019 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerDied","Data":"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd"} Oct 31 03:19:14.249260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.249077 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:19:14.249260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.249193 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.249843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.249809 4556 scope.go:122] "RemoveContainer" containerID="c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd" Oct 31 03:19:14.252408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.252374 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:14.252512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.252476 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:14.252554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.252529 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:14.253637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.253600 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-62bf00\"" Oct 31 03:19:14.301679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.301197 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:19:14.301679 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.301248 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:19:14.327542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.304605 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.327542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.321191 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:14.327542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.327132 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ddac89-e777-4dd8-9d82-3c0e275cd5ae" path="/var/lib/kubelet/pods/f6ddac89-e777-4dd8-9d82-3c0e275cd5ae/volumes" Oct 31 03:19:14.335277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.335187 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.335277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.335246 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.335541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.335283 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cww7h\" (UniqueName: \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.335541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.335480 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrnlc\" (UniqueName: \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.412923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.412865 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-62bf00-dockercfg-sc2gz\"" Oct 31 03:19:14.441538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.441484 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.443261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.443231 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.444553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.444527 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cww7h\" (UniqueName: \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.445536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.442798 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.445735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.444378 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.446358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.446320 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrnlc\" (UniqueName: \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.617460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.615100 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrnlc\" (UniqueName: \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\") pod \"payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.648822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.648788 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:14.664147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.663905 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cww7h\" (UniqueName: \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\") pod \"payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:14.687234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:14.687201 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:15.056793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.056758 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:19:15.059599 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:15.059563 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedc3cc2_fb07_49bb_a553_3283341edeaf.slice/crio-4746679d7829d20a15f436232f9eb815c537b5945dcbe991459e68a6b9685897 WatchSource:0}: Error finding container 4746679d7829d20a15f436232f9eb815c537b5945dcbe991459e68a6b9685897: Status 404 returned error can't find the container with id 4746679d7829d20a15f436232f9eb815c537b5945dcbe991459e68a6b9685897 Oct 31 03:19:15.072352 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:15.072319 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba3de21_6f99_4686_838c_4b35ea81602c.slice/crio-e3bf3b79bb29c9d516effccb4d873988e3af8d6daff67ea8560e4c064810d2ec WatchSource:0}: Error finding container e3bf3b79bb29c9d516effccb4d873988e3af8d6daff67ea8560e4c064810d2ec: Status 404 returned error can't find the container with id e3bf3b79bb29c9d516effccb4d873988e3af8d6daff67ea8560e4c064810d2ec Oct 31 03:19:15.080816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.080765 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:19:15.183155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.183092 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:15.266470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.263532 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" event={"ID":"dedc3cc2-fb07-49bb-a553-3283341edeaf","Type":"ContainerStarted","Data":"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8"} Oct 31 03:19:15.266470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.263628 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" event={"ID":"dedc3cc2-fb07-49bb-a553-3283341edeaf","Type":"ContainerStarted","Data":"4746679d7829d20a15f436232f9eb815c537b5945dcbe991459e68a6b9685897"} Oct 31 03:19:15.266470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.264954 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:15.285325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.285284 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerStarted","Data":"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2"} Oct 31 03:19:15.287649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.287539 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" podStartSLOduration=1.287483456 podStartE2EDuration="1.287483456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:15.281719522 +0000 UTC m=+1433.136736655" watchObservedRunningTime="2026-10-31 03:19:15.287483456 +0000 UTC m=+1433.142500599" Oct 31 03:19:15.297782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.297736 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" event={"ID":"8ba3de21-6f99-4686-838c-4b35ea81602c","Type":"ContainerStarted","Data":"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16"} Oct 31 03:19:15.298031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.298006 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" event={"ID":"8ba3de21-6f99-4686-838c-4b35ea81602c","Type":"ContainerStarted","Data":"e3bf3b79bb29c9d516effccb4d873988e3af8d6daff67ea8560e4c064810d2ec"} Oct 31 03:19:15.298607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.298579 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:15.334022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.333959 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" podStartSLOduration=1.333942734 podStartE2EDuration="1.333942734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:15.331894548 +0000 UTC m=+1433.186911689" watchObservedRunningTime="2026-10-31 03:19:15.333942734 +0000 UTC m=+1433.188959872" Oct 31 03:19:15.728384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.728343 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:19:15.729460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.729375 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:15.729636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.729496 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:15.803959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.803881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:15.938621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:15.938575 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:15.941785 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:15.941745 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:19:16.124821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.124701 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:16.219915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.219884 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:19:16.729730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.729690 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:16.730057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.730019 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:16.969696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.968859 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" podUID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" containerName="payload-processing" containerID="cri-o://65537a73aba0194b1698904b93881d047eb4bc5d867ea0466411def835df7101" gracePeriod=25 Oct 31 03:19:17.010888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:16.991564 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" podUID="28b17cad-376e-4e85-8477-e153e058a644" containerName="payload-pre-processing" containerID="cri-o://0b53cbcb6815904a97a380d1aa71808d994afd4b5c7f8bab535b24842a5144d2" gracePeriod=25 Oct 31 03:19:17.086949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.086865 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.339044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.339007 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.409612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.409562 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:19:17.446788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.446659 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:17.455821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.455788 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7269ae\"" Oct 31 03:19:17.514875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.490871 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7269ae-dockercfg-mdhqf\"" Oct 31 03:19:17.514875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.503663 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:19:17.546300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.545373 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:17.679306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.679247 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.897451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.887183 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:17.897451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.887245 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:17.940733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.940522 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzgl\" (UniqueName: \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:17.953596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.953310 4556 generic.go:356] "Generic (PLEG): container finished" podID="28b17cad-376e-4e85-8477-e153e058a644" containerID="0b53cbcb6815904a97a380d1aa71808d994afd4b5c7f8bab535b24842a5144d2" exitCode=0 Oct 31 03:19:17.953596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.953530 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" event={"ID":"28b17cad-376e-4e85-8477-e153e058a644","Type":"ContainerDied","Data":"0b53cbcb6815904a97a380d1aa71808d994afd4b5c7f8bab535b24842a5144d2"} Oct 31 03:19:17.955988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.955950 4556 generic.go:356] "Generic (PLEG): container finished" podID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" containerID="65537a73aba0194b1698904b93881d047eb4bc5d867ea0466411def835df7101" exitCode=0 Oct 31 03:19:17.957117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.957091 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:19:17.959149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.956156 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" event={"ID":"84e4d2b5-2050-4873-b69c-beafc19b9f0e","Type":"ContainerDied","Data":"65537a73aba0194b1698904b93881d047eb4bc5d867ea0466411def835df7101"} Oct 31 03:19:17.959149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.958590 4556 generic.go:356] "Generic (PLEG): container finished" podID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerID="d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5" exitCode=0 Oct 31 03:19:17.959149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.958662 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerDied","Data":"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5"} Oct 31 03:19:17.959805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.959780 4556 scope.go:122] "RemoveContainer" containerID="d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5" Oct 31 03:19:17.962014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.961992 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:17.962166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.962152 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:17.962276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:17.962264 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:19:18.021209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.021174 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:19:18.048762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.048714 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\") pod \"28b17cad-376e-4e85-8477-e153e058a644\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " Oct 31 03:19:18.049252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.049231 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\" (UniqueName: \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\") pod \"28b17cad-376e-4e85-8477-e153e058a644\" (UID: \"28b17cad-376e-4e85-8477-e153e058a644\") " Oct 31 03:19:18.049928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.049898 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:18.050474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.050452 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzgl\" (UniqueName: \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:18.051325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.051301 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume" pod "28b17cad-376e-4e85-8477-e153e058a644" (UID: "28b17cad-376e-4e85-8477-e153e058a644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:18.057493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.057462 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:18.061346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.061311 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57" pod "28b17cad-376e-4e85-8477-e153e058a644" (UID: "28b17cad-376e-4e85-8477-e153e058a644"). InnerVolumeSpecName "kube-api-access-lvz57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:18.111576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.111545 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzgl\" (UniqueName: \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\") pod \"payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:18.117092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.117051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:18.152161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.152123 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\" (UniqueName: \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\") pod \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " Oct 31 03:19:18.152491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.152465 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\") pod \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\" (UID: \"84e4d2b5-2050-4873-b69c-beafc19b9f0e\") " Oct 31 03:19:18.152855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.152829 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume" pod "84e4d2b5-2050-4873-b69c-beafc19b9f0e" (UID: "84e4d2b5-2050-4873-b69c-beafc19b9f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:18.153785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.153758 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e4d2b5-2050-4873-b69c-beafc19b9f0e-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:18.153926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.153911 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b17cad-376e-4e85-8477-e153e058a644-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:18.154026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.154012 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvz57\" (UniqueName: \"kubernetes.io/projected/28b17cad-376e-4e85-8477-e153e058a644-kube-api-access-lvz57\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:18.159289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.159258 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c" pod "84e4d2b5-2050-4873-b69c-beafc19b9f0e" (UID: "84e4d2b5-2050-4873-b69c-beafc19b9f0e"). InnerVolumeSpecName "kube-api-access-6x52c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:18.258612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.258567 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x52c\" (UniqueName: \"kubernetes.io/projected/84e4d2b5-2050-4873-b69c-beafc19b9f0e-kube-api-access-6x52c\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:18.258811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.258721 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:18.627228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.627185 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:19:18.632290 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:18.632255 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d225d9d_9b90_49ba_9ad1_ffbf1fd56a4f.slice/crio-f371df8a6069c568fc328d4dab2ac54093aa7f5a69f814ec5b7e344fff350057 WatchSource:0}: Error finding container f371df8a6069c568fc328d4dab2ac54093aa7f5a69f814ec5b7e344fff350057: Status 404 returned error can't find the container with id f371df8a6069c568fc328d4dab2ac54093aa7f5a69f814ec5b7e344fff350057 Oct 31 03:19:18.729266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.729224 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:18.729470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.729294 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:18.800323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.800276 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:19:18.826829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.824838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:19.000497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.998884 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" event={"ID":"84e4d2b5-2050-4873-b69c-beafc19b9f0e","Type":"ContainerDied","Data":"7ed73ebc7151ae1b4359c59d4a83d734592f22d8dc9ea60c2853579cd4c0c506"} Oct 31 03:19:19.000497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.998963 4556 scope.go:122] "RemoveContainer" containerID="65537a73aba0194b1698904b93881d047eb4bc5d867ea0466411def835df7101" Oct 31 03:19:19.000497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:18.999139 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj" Oct 31 03:19:19.052334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.051127 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" event={"ID":"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f","Type":"ContainerStarted","Data":"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72"} Oct 31 03:19:19.052334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.051194 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" event={"ID":"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f","Type":"ContainerStarted","Data":"f371df8a6069c568fc328d4dab2ac54093aa7f5a69f814ec5b7e344fff350057"} Oct 31 03:19:19.052334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.052243 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:19.056343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.056318 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerStarted","Data":"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf"} Oct 31 03:19:19.059299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.059275 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" event={"ID":"28b17cad-376e-4e85-8477-e153e058a644","Type":"ContainerDied","Data":"53ca1671b8c07b65ebae942c262f1fdfdced42063c9e7bd9bbb68ea4e26dc5e1"} Oct 31 03:19:19.059435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.059366 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l" Oct 31 03:19:19.060336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.060317 4556 scope.go:122] "RemoveContainer" containerID="0b53cbcb6815904a97a380d1aa71808d994afd4b5c7f8bab535b24842a5144d2" Oct 31 03:19:19.066861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.066833 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:19:19.121864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.121828 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6d1a8e-5854bc4649-2vfdj"] Oct 31 03:19:19.247999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.247965 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:19:19.298580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.298477 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6d1a8e-57b7848db7-kkw5l"] Oct 31 03:19:19.322440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.320513 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" podStartSLOduration=2.320490065 podStartE2EDuration="2.320490065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:19.221789752 +0000 UTC m=+1437.076806905" watchObservedRunningTime="2026-10-31 03:19:19.320490065 +0000 UTC m=+1437.175507208" Oct 31 03:19:19.666471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.666384 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:19:19.666993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.666697 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:19:19.666993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.666781 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:19:19.729262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.728804 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:19.729262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.728864 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:19.947951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:19.947913 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:20.264373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.264211 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b17cad-376e-4e85-8477-e153e058a644" path="/var/lib/kubelet/pods/28b17cad-376e-4e85-8477-e153e058a644/volumes" Oct 31 03:19:20.264954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.264933 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" path="/var/lib/kubelet/pods/84e4d2b5-2050-4873-b69c-beafc19b9f0e/volumes" Oct 31 03:19:20.391584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.391533 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:20.666086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.666050 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:19:20.666282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.666110 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:19:20.729778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.729689 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:19:20.729778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.729751 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:19:20.905464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:20.904949 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:21.665685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.665644 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:19:21.665924 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.665719 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:19:21.725792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.725747 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:21.847547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.844338 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:19:21.847547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.846375 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:19:21.850279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.849666 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:19:21.858762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:21.858727 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.188547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.184823 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.346580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.346541 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.465974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.465875 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:19:22.466186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.466156 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" containerID="cri-o://ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf" gracePeriod=30 Oct 31 03:19:22.548582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.548541 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:22.750486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:22.749553 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:23.086498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:23.084288 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:23.509205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:23.509164 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:24.530753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:24.530645 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:25.571239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:25.571203 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:26.667992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:26.667962 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:19:26.810799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:26.810707 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:27.562888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:27.562853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:27.846921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:27.846833 4556 scope.go:122] "RemoveContainer" containerID="da459e309c93fe7df5a9bd2bb974a2eeeca728185ca682bfc3681cf8bb9e8dfd" Oct 31 03:19:27.858637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:27.858615 4556 scope.go:122] "RemoveContainer" containerID="ca73e6fc4cf0ed8df770a6d3d8b1faf3cead38524097c1a8ff582ca973a42746" Oct 31 03:19:27.959052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:27.959014 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:19:27.962552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:27.962511 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:19:28.098979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.098893 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:28.230778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.230741 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:19:28.231814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231790 4556 cpu_manager.go:414] "containerMap: removing container" podUID="28b17cad-376e-4e85-8477-e153e058a644" containerName="payload-pre-processing" Oct 31 03:19:28.231814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231816 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b17cad-376e-4e85-8477-e153e058a644" containerName="payload-pre-processing" Oct 31 03:19:28.231960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231837 4556 cpu_manager.go:414] "containerMap: removing container" podUID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" containerName="payload-processing" Oct 31 03:19:28.231960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231844 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" containerName="payload-processing" Oct 31 03:19:28.232039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231967 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b17cad-376e-4e85-8477-e153e058a644" containerName="payload-pre-processing" Oct 31 03:19:28.232039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.231980 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e4d2b5-2050-4873-b69c-beafc19b9f0e" containerName="payload-processing" Oct 31 03:19:28.237501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.237471 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.246383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.246348 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:19:28.309704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.309665 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-62bf00-serving-cert\"" Oct 31 03:19:28.378806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.378723 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:28.399384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.399344 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.399588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.399476 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrlq\" (UniqueName: \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.399636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.399589 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.399821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.399802 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.501584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.501530 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.501818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.501692 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.501818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.501788 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrlq\" (UniqueName: \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.501959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.501834 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.502782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.502753 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.504659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.504636 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.506336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.506308 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:28.507989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.507965 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.524005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.523551 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrlq\" (UniqueName: \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\") pod \"maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.574692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.574631 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:28.895522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:28.889502 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:19:29.098970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.098929 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.222404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.222366 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.318532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.318498 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" event={"ID":"ffa98cdf-5d23-4e37-8c63-6e8619231161","Type":"ContainerStarted","Data":"823716230d371a57101eb1bcd9c82ea65f5d206b4d6a0c88bffbe73981cd391d"} Oct 31 03:19:29.355990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.355950 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:29.413075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.412808 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:19:29.413075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.412902 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:19:29.413484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.413106 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" containerID="cri-o://6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90" gracePeriod=30 Oct 31 03:19:29.413484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.413402 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" containerID="cri-o://436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802" gracePeriod=30 Oct 31 03:19:29.431107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.431072 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:19:29.431343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.431305 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" containerID="cri-o://44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab" gracePeriod=30 Oct 31 03:19:29.598145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:29.597956 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:30.106129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.106100 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:19:30.170335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.170297 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:30.248703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.248666 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.248911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.248752 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.248911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.248872 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.248971 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.249043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.248983 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249065 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249088 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.249165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249114 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249115 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.249355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249211 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249226 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.249355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249300 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.249598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.249402 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\") pod \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\" (UID: \"34191e62-c455-40b4-83cb-5d7a2b3a6b85\") " Oct 31 03:19:30.250963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.250857 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.250963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.250887 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.250963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.250902 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.250963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.250916 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.252009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.251975 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:30.252131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.252036 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:30.252363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.252340 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "kube-api-access-6dnlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:30.252561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.252449 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:30.255588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.255559 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy" pod "34191e62-c455-40b4-83cb-5d7a2b3a6b85" (UID: "34191e62-c455-40b4-83cb-5d7a2b3a6b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:30.328007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.327972 4556 generic.go:356] "Generic (PLEG): container finished" podID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerID="ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf" exitCode=0 Oct 31 03:19:30.328192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.328046 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" Oct 31 03:19:30.328192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.328050 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerDied","Data":"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf"} Oct 31 03:19:30.328192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.328113 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59" event={"ID":"34191e62-c455-40b4-83cb-5d7a2b3a6b85","Type":"ContainerDied","Data":"27fc67904cae0126f350b5800b2a6c5cabf98a7f8f49db1c7321eed7b837dfca"} Oct 31 03:19:30.328192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.328160 4556 scope.go:122] "RemoveContainer" containerID="ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf" Oct 31 03:19:30.330330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.330240 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" event={"ID":"ffa98cdf-5d23-4e37-8c63-6e8619231161","Type":"ContainerStarted","Data":"f0c77ae6c4a6dcf1745a5914a663a687675ad338d91a373ea2338cddab07861c"} Oct 31 03:19:30.330535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.330406 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:30.341764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.341640 4556 scope.go:122] "RemoveContainer" containerID="d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5" Oct 31 03:19:30.348644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.348611 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:19:30.352724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.352678 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.352724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.352711 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.352724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.352725 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.352927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.352736 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnlt\" (UniqueName: \"kubernetes.io/projected/34191e62-c455-40b4-83cb-5d7a2b3a6b85-kube-api-access-6dnlt\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.352927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.352745 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34191e62-c455-40b4-83cb-5d7a2b3a6b85-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:30.353282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.353257 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-6d1a8e-openshift-default-dbbf5f755-6nq59"] Oct 31 03:19:30.355793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.355766 4556 scope.go:122] "RemoveContainer" containerID="ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf" Oct 31 03:19:30.356224 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:30.356200 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf\": container with ID starting with ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf not found: ID does not exist" containerID="ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf" Oct 31 03:19:30.356329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.356231 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf"} err="failed to get container status \"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf\": rpc error: code = NotFound desc = could not find container \"ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf\": container with ID starting with ad9022d1be4f40d00689bdfccca0cbd82d1cd4248a54275f87485e4114824dcf not found: ID does not exist" Oct 31 03:19:30.356329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.356254 4556 scope.go:122] "RemoveContainer" containerID="d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5" Oct 31 03:19:30.356672 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:30.356643 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5\": container with ID starting with d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5 not found: ID does not exist" containerID="d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5" Oct 31 03:19:30.356784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.356679 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5"} err="failed to get container status \"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5\": rpc error: code = NotFound desc = could not find container \"d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5\": container with ID starting with d69b28e7131c5f05a857f1d4ee4f5149cda80add3ddcabf6dced108355b19ac5 not found: ID does not exist" Oct 31 03:19:30.368705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.368023 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" podStartSLOduration=1.9151702099999999 podStartE2EDuration="2.368004101s" totalImagesPullingTime="452.833891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:30.366953505 +0000 UTC m=+1448.221970646" watchObservedRunningTime="2026-10-31 03:19:30.368004101 +0000 UTC m=+1448.223021255" Oct 31 03:19:30.499489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:30.499324 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.130865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.130818 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:19:31.132074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132051 4556 cpu_manager.go:414] "containerMap: removing container" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.132142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132077 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.132142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132093 4556 cpu_manager.go:414] "containerMap: removing container" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.132142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132101 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.132285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132266 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.132347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.132287 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" containerName="istio-proxy" Oct 31 03:19:31.145225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.145193 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.146206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.146181 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:19:31.154329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.154299 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7269ae-serving-cert\"" Oct 31 03:19:31.215875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.215834 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.243395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.243354 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:19:31.270673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.270627 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.271640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.271001 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.271640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.271204 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm96q\" (UniqueName: \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.271640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.271300 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.350078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.349931 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.373030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.372996 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm96q\" (UniqueName: \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.373232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.373153 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.373468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.373448 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.373589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.373569 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.374362 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:31.374340 4556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-serving-cert: secret "maas-api-e2e-inf-b-7269ae-serving-cert" not found Oct 31 03:19:31.374440 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:31.374432 4556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls podName:9840d12a-89c6-44d4-976d-5b32edd8a23d nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.874393794 +0000 UTC m=+1449.729410934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls") pod "maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" (UID: "9840d12a-89c6-44d4-976d-5b32edd8a23d") : secret "maas-api-e2e-inf-b-7269ae-serving-cert" not found Oct 31 03:19:31.374981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.374957 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.378401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.378376 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.385814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.385748 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm96q\" (UniqueName: \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.661446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.658339 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:31.885302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.885212 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:31.888053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:31.888023 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") pod \"maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:32.062196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.062099 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:32.065076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.065047 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:32.206577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.206540 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:32.271810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.271766 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34191e62-c455-40b4-83cb-5d7a2b3a6b85" path="/var/lib/kubelet/pods/34191e62-c455-40b4-83cb-5d7a2b3a6b85/volumes" Oct 31 03:19:32.356589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.356552 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:19:32.357396 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:19:32.357363 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9840d12a_89c6_44d4_976d_5b32edd8a23d.slice/crio-b22688eaeda44b98a8150094a0c69bb08ac03a22b9792e9fe6ad624f85a09da1 WatchSource:0}: Error finding container b22688eaeda44b98a8150094a0c69bb08ac03a22b9792e9fe6ad624f85a09da1: Status 404 returned error can't find the container with id b22688eaeda44b98a8150094a0c69bb08ac03a22b9792e9fe6ad624f85a09da1 Oct 31 03:19:32.475927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.475887 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:32.844964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:32.844927 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:33.317608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.317559 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:33.378726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.378663 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" event={"ID":"9840d12a-89c6-44d4-976d-5b32edd8a23d","Type":"ContainerStarted","Data":"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11"} Oct 31 03:19:33.378946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.378738 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" event={"ID":"9840d12a-89c6-44d4-976d-5b32edd8a23d","Type":"ContainerStarted","Data":"b22688eaeda44b98a8150094a0c69bb08ac03a22b9792e9fe6ad624f85a09da1"} Oct 31 03:19:33.378946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.378800 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:33.400288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.400222 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" podStartSLOduration=1.9080763520000001 podStartE2EDuration="2.400205055s" totalImagesPullingTime="492.128703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:33.396641539 +0000 UTC m=+1451.251658681" watchObservedRunningTime="2026-10-31 03:19:33.400205055 +0000 UTC m=+1451.255222195" Oct 31 03:19:33.708644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.708611 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:33.844382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:33.844349 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:34.557057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.557024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:34.832333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.826174 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.857755 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.857824 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.857906 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858025 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858062 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858188 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858278 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858351 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.858730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.858519 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\") pod \"556f9edf-13be-4654-b987-5efdda7df8a5\" (UID: \"556f9edf-13be-4654-b987-5efdda7df8a5\") " Oct 31 03:19:34.861097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.860542 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.861097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.861028 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.861097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.861054 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:34.861370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.861195 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.861570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.861522 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.867160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.867125 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:34.867710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.867656 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "kube-api-access-djbxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:34.868240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.868202 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.868483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.868451 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo" pod "556f9edf-13be-4654-b987-5efdda7df8a5" (UID: "556f9edf-13be-4654-b987-5efdda7df8a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:34.880440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.877957 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:19:34.945901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.945871 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:19:34.959929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.959894 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.959996 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960051 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960089 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960117 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960159 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960194 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960327 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960371 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.960598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960388 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\") pod \"c41f8be1-e9fc-478f-96fc-29c9bea88868\" (UID: \"c41f8be1-e9fc-478f-96fc-29c9bea88868\") " Oct 31 03:19:34.960762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.960639 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.961079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961000 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.961079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961035 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:34.961434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961351 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961383 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556f9edf-13be-4654-b987-5efdda7df8a5-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961400 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961465 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961484 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556f9edf-13be-4654-b987-5efdda7df8a5-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961499 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961513 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961528 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961542 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961557 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.961600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961572 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.962965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961779 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556f9edf-13be-4654-b987-5efdda7df8a5-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.962965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.961802 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djbxh\" (UniqueName: \"kubernetes.io/projected/556f9edf-13be-4654-b987-5efdda7df8a5-kube-api-access-djbxh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:34.963110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.963081 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:34.963790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.963762 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:34.991680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.991612 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "kube-api-access-lkpnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:34.992069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.992024 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:34.993892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:34.993838 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy" pod "c41f8be1-e9fc-478f-96fc-29c9bea88868" (UID: "c41f8be1-e9fc-478f-96fc-29c9bea88868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.063136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063042 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063105 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063189 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063235 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063323 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063380 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063509 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063616 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.063649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063642 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063643 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:19:35.063753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063647 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.063753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063746 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\") pod \"8c264d64-9dff-40ce-acfe-f3cd0447f968\" (UID: \"8c264d64-9dff-40ce-acfe-f3cd0447f968\") " Oct 31 03:19:35.063918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063755 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.063918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.063887 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.064758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064729 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c264d64-9dff-40ce-acfe-f3cd0447f968-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064762 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064777 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064791 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064806 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkpnk\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-kube-api-access-lkpnk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064823 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41f8be1-e9fc-478f-96fc-29c9bea88868-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064838 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064852 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.064871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064867 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.065554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.064880 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41f8be1-e9fc-478f-96fc-29c9bea88868-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.065715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.065692 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:35.065828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.065777 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "kube-api-access-ztmhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.065828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.065796 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:19:35.066043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.066019 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token" pod "8c264d64-9dff-40ce-acfe-f3cd0447f968" (UID: "8c264d64-9dff-40ce-acfe-f3cd0447f968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:35.167149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.167110 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.167149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.167147 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.167376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.167165 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmhz\" (UniqueName: \"kubernetes.io/projected/8c264d64-9dff-40ce-acfe-f3cd0447f968-kube-api-access-ztmhz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.167376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.167178 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c264d64-9dff-40ce-acfe-f3cd0447f968-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:35.353378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.353208 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:35.403086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.403045 4556 generic.go:356] "Generic (PLEG): container finished" podID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerID="6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90" exitCode=0 Oct 31 03:19:35.403598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.403132 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" Oct 31 03:19:35.403742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.403137 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerDied","Data":"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90"} Oct 31 03:19:35.403845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.403773 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq" event={"ID":"8c264d64-9dff-40ce-acfe-f3cd0447f968","Type":"ContainerDied","Data":"8ecb99c56b38b3e4004a849c67561aae00fb1aa3591af236539c8b7f5ca5c6fc"} Oct 31 03:19:35.403845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.403837 4556 scope.go:122] "RemoveContainer" containerID="6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90" Oct 31 03:19:35.406267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.406240 4556 generic.go:356] "Generic (PLEG): container finished" podID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerID="44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab" exitCode=0 Oct 31 03:19:35.406477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.406308 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" Oct 31 03:19:35.406477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.406330 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" event={"ID":"c41f8be1-e9fc-478f-96fc-29c9bea88868","Type":"ContainerDied","Data":"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab"} Oct 31 03:19:35.406477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.406396 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94" event={"ID":"c41f8be1-e9fc-478f-96fc-29c9bea88868","Type":"ContainerDied","Data":"c8445167808f6fc0bd06ff17de77e2c4207cfbe3abb33a10d044bdcceba2053d"} Oct 31 03:19:35.411154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.410867 4556 generic.go:356] "Generic (PLEG): container finished" podID="556f9edf-13be-4654-b987-5efdda7df8a5" containerID="436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802" exitCode=0 Oct 31 03:19:35.411154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.410954 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" event={"ID":"556f9edf-13be-4654-b987-5efdda7df8a5","Type":"ContainerDied","Data":"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802"} Oct 31 03:19:35.411154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.411035 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" event={"ID":"556f9edf-13be-4654-b987-5efdda7df8a5","Type":"ContainerDied","Data":"cc11e0560f5a34c30e78dfd88ae5f9dd423ff32ed5c8fefd318003f0b2e09ec1"} Oct 31 03:19:35.411154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.411110 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59" Oct 31 03:19:35.441381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.441352 4556 scope.go:122] "RemoveContainer" containerID="3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464" Oct 31 03:19:35.499908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.499871 4556 scope.go:122] "RemoveContainer" containerID="6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90" Oct 31 03:19:35.500282 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:35.500258 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90\": container with ID starting with 6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90 not found: ID does not exist" containerID="6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90" Oct 31 03:19:35.500353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.500290 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90"} err="failed to get container status \"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90\": rpc error: code = NotFound desc = could not find container \"6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90\": container with ID starting with 6e438a3a889553dc6086336b92f42475cd7434ba5a8eb2cee3e7e2829c7dee90 not found: ID does not exist" Oct 31 03:19:35.500353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.500307 4556 scope.go:122] "RemoveContainer" containerID="3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464" Oct 31 03:19:35.500669 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:35.500649 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464\": container with ID starting with 3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464 not found: ID does not exist" containerID="3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464" Oct 31 03:19:35.500766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.500677 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464"} err="failed to get container status \"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464\": rpc error: code = NotFound desc = could not find container \"3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464\": container with ID starting with 3f51e71b5e6d9c30854c4e3887978f4631a51500753f649fe7f162fd4074e464 not found: ID does not exist" Oct 31 03:19:35.500766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.500697 4556 scope.go:122] "RemoveContainer" containerID="44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab" Oct 31 03:19:35.514745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.514712 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:19:35.514886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.514731 4556 scope.go:122] "RemoveContainer" containerID="44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab" Oct 31 03:19:35.515164 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:35.515137 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab\": container with ID starting with 44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab not found: ID does not exist" containerID="44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab" Oct 31 03:19:35.515257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.515176 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab"} err="failed to get container status \"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab\": rpc error: code = NotFound desc = could not find container \"44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab\": container with ID starting with 44c8b82440bcee07ee02e5d45f214b85ca5f2ed65c3d30eb36c738afa6fd3bab not found: ID does not exist" Oct 31 03:19:35.515257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.515200 4556 scope.go:122] "RemoveContainer" containerID="436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802" Oct 31 03:19:35.518832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.518804 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-4lcnq"] Oct 31 03:19:35.528541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.528515 4556 scope.go:122] "RemoveContainer" containerID="436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802" Oct 31 03:19:35.528929 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:35.528900 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802\": container with ID starting with 436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802 not found: ID does not exist" containerID="436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802" Oct 31 03:19:35.529005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.528942 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802"} err="failed to get container status \"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802\": rpc error: code = NotFound desc = could not find container \"436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802\": container with ID starting with 436bf8c964e8c416d1e39bd837e503f8677502314b2b2801c3b825cad4eaa802 not found: ID does not exist" Oct 31 03:19:35.531720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.531689 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:19:35.536180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.536154 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-9fp59"] Oct 31 03:19:35.547296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.547262 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:19:35.551056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:35.551031 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1bfbe55-openshift-default-58766cfb9-dzk94"] Oct 31 03:19:36.261089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.261050 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" path="/var/lib/kubelet/pods/556f9edf-13be-4654-b987-5efdda7df8a5/volumes" Oct 31 03:19:36.261769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.261750 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" path="/var/lib/kubelet/pods/8c264d64-9dff-40ce-acfe-f3cd0447f968/volumes" Oct 31 03:19:36.262232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.262218 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" path="/var/lib/kubelet/pods/c41f8be1-e9fc-478f-96fc-29c9bea88868/volumes" Oct 31 03:19:36.322751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.322672 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:36.362200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.362171 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:19:36.607783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.607646 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:36.758486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:36.758394 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:37.162129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:37.162086 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:37.736628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:37.736582 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:19:37.926657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:37.926602 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:38.067494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:38.066881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.052556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:39.052495 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:39.395734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:39.395653 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:19:39.831335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:39.830981 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:40.574823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:40.574779 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:41.321164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:41.321114 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:42.013567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:42.013459 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:42.441589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:42.441551 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:42.582974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:42.582935 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:42.984098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:42.984052 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" containerID="cri-o://633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b" gracePeriod=45 Oct 31 03:19:43.000529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.000488 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:43.324913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.324884 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:19:43.488598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.488555 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\" (UniqueName: \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.488828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.488811 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.488900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.488872 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.488958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.488906 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.489103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489088 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.489161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489097 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.489161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489124 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\") pod \"3b77c8f0-3fd1-44c5-a139-413e999765a9\" (UID: \"3b77c8f0-3fd1-44c5-a139-413e999765a9\") " Oct 31 03:19:43.489427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489384 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.489501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489396 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.489814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489795 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.489883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489818 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.489883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.489833 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.491006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.490974 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "kube-api-access-lflg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:19:43.491128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.491071 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:19:43.491177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.491126 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm" pod "3b77c8f0-3fd1-44c5-a139-413e999765a9" (UID: "3b77c8f0-3fd1-44c5-a139-413e999765a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:19:43.524572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.521677 4556 generic.go:356] "Generic (PLEG): container finished" podID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerID="633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b" exitCode=0 Oct 31 03:19:43.524572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.522092 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" event={"ID":"3b77c8f0-3fd1-44c5-a139-413e999765a9","Type":"ContainerDied","Data":"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b"} Oct 31 03:19:43.524572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.522144 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" event={"ID":"3b77c8f0-3fd1-44c5-a139-413e999765a9","Type":"ContainerDied","Data":"63ee89c770c11c5bd706d21f80e59572921f23b5c8918004eb0ba979e3ffa22f"} Oct 31 03:19:43.524572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.522168 4556 scope.go:122] "RemoveContainer" containerID="633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b" Oct 31 03:19:43.524572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.522389 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s" Oct 31 03:19:43.538121 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.538099 4556 scope.go:122] "RemoveContainer" containerID="633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b" Oct 31 03:19:43.538520 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:43.538494 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b\": container with ID starting with 633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b not found: ID does not exist" containerID="633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b" Oct 31 03:19:43.538594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.538531 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b"} err="failed to get container status \"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b\": rpc error: code = NotFound desc = could not find container \"633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b\": container with ID starting with 633c08c1a6900ac8f606e8ad20257368941af1d629edbc63dd69a0eaed8f589b not found: ID does not exist" Oct 31 03:19:43.547050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.547015 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:19:43.551255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.551225 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-edb9e6/distinct-2-w0-edb9e6-kserve-7d6bdbd9f7-qhd7s"] Oct 31 03:19:43.591535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.591500 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b77c8f0-3fd1-44c5-a139-413e999765a9-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.591770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.591754 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b77c8f0-3fd1-44c5-a139-413e999765a9-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.591864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.591852 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflg7\" (UniqueName: \"kubernetes.io/projected/3b77c8f0-3fd1-44c5-a139-413e999765a9-kube-api-access-lflg7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:19:43.735568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:43.735526 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:44.261746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:44.261700 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" path="/var/lib/kubelet/pods/3b77c8f0-3fd1-44c5-a139-413e999765a9/volumes" Oct 31 03:19:44.540456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:44.540325 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.698841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:45.698803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:45.939865 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:19:45.939820 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:19:46.525301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:46.525250 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:47.582073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:47.582025 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:48.473831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:48.473792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:48.599842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:48.599808 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:48.948620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:48.948580 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.324450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:49.324313 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.458388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:49.458354 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:49.662975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:49.662884 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:50.328450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:50.328372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:50.460748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:50.460711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:51.165248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:51.165213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:51.295457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:51.295404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:52.148000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:52.147957 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:53.205111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:53.205078 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:53.345444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:53.343030 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:53.911578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:53.911538 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:54.380133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:54.379997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:54.865774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:54.865737 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:54.999633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:54.999597 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:55.380320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:55.380281 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.247781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:56.247742 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:57.196407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:57.196367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:58.009551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:58.009511 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:58.898400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:58.898367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:59.735111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:19:59.735073 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:00.510505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:00.510463 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.100528 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:01.100489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.277444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:01.269377 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.404652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:01.404615 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.716029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:01.715906 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:01.855512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:01.855476 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:02.611682 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:02.611643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:03.487507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:03.487468 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:04.299226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:04.298990 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:04.535395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:04.535341 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.069081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.069037 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:20:05.076049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.075980 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" podUID="48071f9d-c984-450e-b06e-69eef341d749" containerName="maas-api" containerID="cri-o://5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c" gracePeriod=30 Oct 31 03:20:05.162721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.162692 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.342816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.342764 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.373833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.373797 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:20:05.473539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.473493 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.493929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.493890 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\") pod \"48071f9d-c984-450e-b06e-69eef341d749\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " Oct 31 03:20:05.494102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.493983 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\") pod \"48071f9d-c984-450e-b06e-69eef341d749\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " Oct 31 03:20:05.494102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.494026 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\" (UniqueName: \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\") pod \"48071f9d-c984-450e-b06e-69eef341d749\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " Oct 31 03:20:05.494218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.494206 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\") pod \"48071f9d-c984-450e-b06e-69eef341d749\" (UID: \"48071f9d-c984-450e-b06e-69eef341d749\") " Oct 31 03:20:05.494702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.494666 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca" pod "48071f9d-c984-450e-b06e-69eef341d749" (UID: "48071f9d-c984-450e-b06e-69eef341d749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:05.495208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.495179 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48071f9d-c984-450e-b06e-69eef341d749-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:05.496207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.496177 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9" pod "48071f9d-c984-450e-b06e-69eef341d749" (UID: "48071f9d-c984-450e-b06e-69eef341d749"). InnerVolumeSpecName "kube-api-access-zv5d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:05.496292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.496204 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls" pod "48071f9d-c984-450e-b06e-69eef341d749" (UID: "48071f9d-c984-450e-b06e-69eef341d749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:05.496292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.496230 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs" pod "48071f9d-c984-450e-b06e-69eef341d749" (UID: "48071f9d-c984-450e-b06e-69eef341d749"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:05.596658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.596555 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:05.596658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.596592 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48071f9d-c984-450e-b06e-69eef341d749-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:05.596658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.596606 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5d9\" (UniqueName: \"kubernetes.io/projected/48071f9d-c984-450e-b06e-69eef341d749-kube-api-access-zv5d9\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:05.612913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.612877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.806442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.805913 4556 generic.go:356] "Generic (PLEG): container finished" podID="48071f9d-c984-450e-b06e-69eef341d749" containerID="5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c" exitCode=0 Oct 31 03:20:05.806442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.806042 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" event={"ID":"48071f9d-c984-450e-b06e-69eef341d749","Type":"ContainerDied","Data":"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c"} Oct 31 03:20:05.806442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.806090 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" event={"ID":"48071f9d-c984-450e-b06e-69eef341d749","Type":"ContainerDied","Data":"e69412c7c80b83ec82cc2486f5a79f0351697eb818bd1dcfe3584386e5a90929"} Oct 31 03:20:05.806442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.806116 4556 scope.go:122] "RemoveContainer" containerID="5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c" Oct 31 03:20:05.806826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.806521 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8" Oct 31 03:20:05.849584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.849137 4556 scope.go:122] "RemoveContainer" containerID="5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c" Oct 31 03:20:05.849917 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:05.849835 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c\": container with ID starting with 5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c not found: ID does not exist" containerID="5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c" Oct 31 03:20:05.849917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.849880 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c"} err="failed to get container status \"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c\": rpc error: code = NotFound desc = could not find container \"5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c\": container with ID starting with 5854b271e767478f6f8720bd869ef4150fdc23e04ed40981fbc75ed8b2833e8c not found: ID does not exist" Oct 31 03:20:05.915883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.915515 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:20:05.915883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.915589 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fd580-5949fb895c-p8rc8"] Oct 31 03:20:05.991369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:05.991321 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.260171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:06.260132 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48071f9d-c984-450e-b06e-69eef341d749" path="/var/lib/kubelet/pods/48071f9d-c984-450e-b06e-69eef341d749/volumes" Oct 31 03:20:06.859249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:06.859210 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:07.628698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:07.628649 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:07.760123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:07.760090 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:07.896517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:07.896048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.317572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:08.317536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.725754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:08.725721 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:08.861823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:08.861782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:09.250244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:09.250208 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:09.719947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:09.719905 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:09.988233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:09.988151 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:10.135439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:10.123920 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:10.336129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:10.336022 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:20:10.337297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:10.337261 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" containerID="cri-o://4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b" gracePeriod=30 Oct 31 03:20:10.384678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:10.384641 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:11.052904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:11.052860 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:12.051438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:12.045060 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:12.315835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:12.315728 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:13.001396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.001358 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:13.173066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.173024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:13.668532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.668487 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:20:13.699441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.669890 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:13.735862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.735827 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:13.736203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736184 4556 cpu_manager.go:414] "containerMap: removing container" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" Oct 31 03:20:13.736333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736319 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736594 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736732 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736780 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736789 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736808 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736816 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736853 4556 cpu_manager.go:414] "containerMap: removing container" podUID="48071f9d-c984-450e-b06e-69eef341d749" containerName="maas-api" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.736862 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48071f9d-c984-450e-b06e-69eef341d749" containerName="maas-api" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.737094 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41f8be1-e9fc-478f-96fc-29c9bea88868" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.737113 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.737125 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b77c8f0-3fd1-44c5-a139-413e999765a9" containerName="main" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.737139 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="48071f9d-c984-450e-b06e-69eef341d749" containerName="maas-api" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.737152 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="556f9edf-13be-4654-b987-5efdda7df8a5" containerName="istio-proxy" Oct 31 03:20:13.759506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.743405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:13.812483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.790774 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.867669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.867627 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bbbc81-openshift-default-dockercfg-25vw2\"" Oct 31 03:20:13.989838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.989800 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmlm\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.990144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.990099 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.990399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.990380 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.990624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.990607 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.990795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.990777 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.990975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.990957 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.991165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.991143 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.991358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.991341 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:13.991551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:13.991534 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.024190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.024157 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.097589 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.146610 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.146963 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmlm\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147006 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147106 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147200 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147255 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147333 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.147390 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.148112 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.098222 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.156456 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.157235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.156750 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.158074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.157405 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.382500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.378610 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.448442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.431404 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.682832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.682759 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.683672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.683642 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmlm\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.743436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.718056 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:14.745253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.743759 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.765450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.745075 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.849440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.847554 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.863145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.852780 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:20:14.905473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.888125 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:14.905473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:14.896387 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\") pod \"e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:15.048270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.048150 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:15.131741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.131697 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.133309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.133275 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd" podUID="ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 31 03:20:15.135898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.135864 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.136178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.136148 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 31 03:20:15.161672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.139026 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:15.283454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.277530 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.283454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.279203 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.542441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.496447 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.542441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.498482 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:20:15.777440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.767724 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.789399 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.789363 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:20:15.826368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.824997 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.826368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.825690 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:20:15.846874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.846720 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:20:15.914470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.907298 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.919388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.909806 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.925831 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.920140 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:20:15.939317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.887613 4556 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:15.942508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:15.942478 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:20:16.158607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.158573 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:20:16.223439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.220780 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:20:16.223439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.220870 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:20:16.227503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.226920 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.304625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.227768 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:16.365089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.361617 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:20:16.378494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.378460 4556 patch_prober.go:29] interesting pod/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 31 03:20:16.383999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.383923 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 31 03:20:16.427999 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:16.425302 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:20:16.436507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.432054 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:20:16.436507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.432125 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:20:16.436507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.432162 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:20:16.453631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.453596 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:20:16.506398 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:16.506349 4556 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071bbc6_f4be_4a6c_a499_8eb27cc71926.slice/crio-4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071bbc6_f4be_4a6c_a499_8eb27cc71926.slice/crio-conmon-4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:20:16.579476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.579051 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:16.688588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.688548 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:20:16.713141 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:20:16.713097 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05d12c4_a22d_4420_afde_9c49f1361d31.slice/crio-30f58b635655d23834ab12716dec4f7361f9b55ca8f063e2f2039765e5168b6a WatchSource:0}: Error finding container 30f58b635655d23834ab12716dec4f7361f9b55ca8f063e2f2039765e5168b6a: Status 404 returned error can't find the container with id 30f58b635655d23834ab12716dec4f7361f9b55ca8f063e2f2039765e5168b6a Oct 31 03:20:16.717377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.717331 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:16.717661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.717631 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:16.717827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.717806 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:16.747905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.747871 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:20:16.854002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.853967 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854080 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854161 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854203 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854236 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854387 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854434 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854480 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.854605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.854582 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\") pod \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\" (UID: \"e071bbc6-f4be-4a6c-a499-8eb27cc71926\") " Oct 31 03:20:16.856426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.855809 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:16.857089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.856846 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:16.857669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.857626 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:16.858100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.857997 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:16.858100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.858068 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:16.859879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.859850 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.859879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.859877 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.860064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.859894 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.860064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.859909 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.860064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.859923 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.860876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.860849 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "kube-api-access-g7ct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:16.863238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.862752 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:16.866534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.866496 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:16.866534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.866517 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token" pod "e071bbc6-f4be-4a6c-a499-8eb27cc71926" (UID: "e071bbc6-f4be-4a6c-a499-8eb27cc71926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:16.922960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.916765 4556 generic.go:356] "Generic (PLEG): container finished" podID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerID="4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b" exitCode=0 Oct 31 03:20:16.922960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.917083 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerDied","Data":"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b"} Oct 31 03:20:16.922960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.917144 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" event={"ID":"e071bbc6-f4be-4a6c-a499-8eb27cc71926","Type":"ContainerDied","Data":"33502efd8427eb600998465eef0b21fa87f5fe377cbaf8654f8654ed822d0ced"} Oct 31 03:20:16.922960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.917178 4556 scope.go:122] "RemoveContainer" containerID="4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b" Oct 31 03:20:16.922960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.917377 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj" Oct 31 03:20:16.941663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.937669 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerStarted","Data":"30f58b635655d23834ab12716dec4f7361f9b55ca8f063e2f2039765e5168b6a"} Oct 31 03:20:16.956760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.956723 4556 scope.go:122] "RemoveContainer" containerID="c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919" Oct 31 03:20:16.963027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.962989 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.963344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.963320 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.963507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.963489 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ct6\" (UniqueName: \"kubernetes.io/projected/e071bbc6-f4be-4a6c-a499-8eb27cc71926-kube-api-access-g7ct6\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:16.963628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:16.963615 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e071bbc6-f4be-4a6c-a499-8eb27cc71926-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:17.015356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.015310 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:20:17.020685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.020468 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-edb9e6-openshift-default-58cd8859f-r9lwj"] Oct 31 03:20:17.020810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.020783 4556 scope.go:122] "RemoveContainer" containerID="4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b" Oct 31 03:20:17.021288 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:17.021248 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b\": container with ID starting with 4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b not found: ID does not exist" containerID="4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b" Oct 31 03:20:17.021378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.021298 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b"} err="failed to get container status \"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b\": rpc error: code = NotFound desc = could not find container \"4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b\": container with ID starting with 4445ccef8bd449c877fd37e7532b420d9cf5c5815229fc3b82517a89339afc9b not found: ID does not exist" Oct 31 03:20:17.021378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.021318 4556 scope.go:122] "RemoveContainer" containerID="c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919" Oct 31 03:20:17.021716 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:17.021699 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919\": container with ID starting with c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919 not found: ID does not exist" containerID="c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919" Oct 31 03:20:17.021787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.021720 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919"} err="failed to get container status \"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919\": rpc error: code = NotFound desc = could not find container \"c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919\": container with ID starting with c1448e8420899821de55a6fc4b019465a2c049a5af985765636b3ef9f78fb919 not found: ID does not exist" Oct 31 03:20:17.632047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.631995 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:17.833169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.833070 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:17.953221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.953189 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerStarted","Data":"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf"} Oct 31 03:20:17.962180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.962036 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:20:17.962400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:17.962290 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" containerID="cri-o://2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2" gracePeriod=30 Oct 31 03:20:18.022763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.022702 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podStartSLOduration=5.022681699 podStartE2EDuration="5.022681699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:18.002848183 +0000 UTC m=+1495.857865325" watchObservedRunningTime="2026-10-31 03:20:18.022681699 +0000 UTC m=+1495.877698842" Oct 31 03:20:18.061331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.061291 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:18.144436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.139789 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:18.157818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.157780 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:18.312493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.312200 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" path="/var/lib/kubelet/pods/e071bbc6-f4be-4a6c-a499-8eb27cc71926/volumes" Oct 31 03:20:18.965085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.965055 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:18.966186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:18.966162 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:20:19.376628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:19.376282 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:20.536282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:20.536239 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:21.854317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:21.854258 4556 patch_prober.go:29] interesting pod/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:20:21.855110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:21.854429 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:20:21.976558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:21.970697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:22.248440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:22.248164 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:22.541885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:22.541763 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:23.370032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.369996 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:23.395814 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:23.395778 4556 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137b2f24_7930_414e_acec_df30d6a1f713.slice/crio-conmon-a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:20:23.412236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.412199 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:20:23.515097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515006 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515085 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515120 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515166 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515202 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515236 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515273 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515313 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515342 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\") pod \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\" (UID: \"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b\") " Oct 31 03:20:23.515886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515356 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:23.515886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515380 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:23.515886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515538 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:23.515886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515554 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:23.515886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.515880 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:23.516282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.516261 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.516327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.516289 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.516327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.516304 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.516327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.516319 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.516471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.516333 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.517719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.517688 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:20:23.518056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.518029 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "kube-api-access-6h6t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:23.518167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.518086 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:23.519808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.519777 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token" pod "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" (UID: "ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:23.584530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.584455 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:20:23.585614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585586 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:23.585614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585610 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:23.585825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585634 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:23.585825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585642 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:23.585940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585853 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:23.585940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585865 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c264d64-9dff-40ce-acfe-f3cd0447f968" containerName="istio-proxy" Oct 31 03:20:23.585940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585875 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:23.585940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585887 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:23.585940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.585895 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:23.594340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.591760 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.595517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.595199 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:20:23.625460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.625396 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xhx\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.625748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.625723 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.625920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.625905 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.626286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.626260 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.626848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.626822 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.627012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.626993 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.627193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.627175 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.627338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.627320 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.627573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.627554 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.627829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.627810 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.627952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.627936 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.628053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.628039 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.628145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.628133 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6t7\" (UniqueName: \"kubernetes.io/projected/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b-kube-api-access-6h6t7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:23.729371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729321 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729371 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729433 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729474 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729563 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729612 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xhx\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729647 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729721 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.729854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729756 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.730075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.729982 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.730075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.730021 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.730316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.730290 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.730406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.730358 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.730638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.730618 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.732330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.732307 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.732502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.732484 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.743884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.743855 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.744016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.743923 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xhx\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:23.933916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:23.933854 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:24.017637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.017596 4556 generic.go:356] "Generic (PLEG): container finished" podID="137b2f24-7930-414e-acec-df30d6a1f713" containerID="a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8" exitCode=0 Oct 31 03:20:24.018161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.017879 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerDied","Data":"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8"} Oct 31 03:20:24.018907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.018889 4556 scope.go:122] "RemoveContainer" containerID="a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8" Oct 31 03:20:24.023548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.022207 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.023548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.022306 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.023548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.022356 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.029476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.029291 4556 generic.go:356] "Generic (PLEG): container finished" podID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerID="2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2" exitCode=0 Oct 31 03:20:24.031599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.029695 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerDied","Data":"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2"} Oct 31 03:20:24.031599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.029766 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" event={"ID":"ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b","Type":"ContainerDied","Data":"0a44d02d6a47e1169d3d014addbc155c054f49dc0eb29a5e5b7f0eec986e4224"} Oct 31 03:20:24.031599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.029801 4556 scope.go:122] "RemoveContainer" containerID="2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2" Oct 31 03:20:24.031599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.029990 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw" Oct 31 03:20:24.068041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.067910 4556 scope.go:122] "RemoveContainer" containerID="c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd" Oct 31 03:20:24.080105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.080071 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:20:24.084839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.084804 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fd580-openshift-default-5459bd4cd9-c8bsw"] Oct 31 03:20:24.100878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.100686 4556 scope.go:122] "RemoveContainer" containerID="2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2" Oct 31 03:20:24.101097 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:24.101072 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2\": container with ID starting with 2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2 not found: ID does not exist" containerID="2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2" Oct 31 03:20:24.101167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.101109 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2"} err="failed to get container status \"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2\": rpc error: code = NotFound desc = could not find container \"2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2\": container with ID starting with 2d1cb550acad85365882ab692e7d5abe7f01e91937d65ab364cf2863dc09d3e2 not found: ID does not exist" Oct 31 03:20:24.101167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.101133 4556 scope.go:122] "RemoveContainer" containerID="c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd" Oct 31 03:20:24.101599 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:24.101477 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd\": container with ID starting with c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd not found: ID does not exist" containerID="c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd" Oct 31 03:20:24.101599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.101510 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd"} err="failed to get container status \"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd\": rpc error: code = NotFound desc = could not find container \"c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd\": container with ID starting with c3fc315141606ccb6be02128cdd6ea4503f0655251f3d999c5ae2d03d20e6fdd not found: ID does not exist" Oct 31 03:20:24.235811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.235778 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:20:24.236002 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:20:24.235965 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ab0f9_98e5_42b6_95fb_86317586e85c.slice/crio-6426476196f5c381ba13ac2f91f1b18df52ce3e99c55bf64a759221073ac987d WatchSource:0}: Error finding container 6426476196f5c381ba13ac2f91f1b18df52ce3e99c55bf64a759221073ac987d: Status 404 returned error can't find the container with id 6426476196f5c381ba13ac2f91f1b18df52ce3e99c55bf64a759221073ac987d Oct 31 03:20:24.239209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.239146 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.240131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.239238 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.240131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.239283 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:24.262597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.262561 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" path="/var/lib/kubelet/pods/ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b/volumes" Oct 31 03:20:24.322349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:24.322303 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:25.039239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.039200 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerStarted","Data":"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff"} Oct 31 03:20:25.041372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.041340 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" event={"ID":"bd3ab0f9-98e5-42b6-95fb-86317586e85c","Type":"ContainerStarted","Data":"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976"} Oct 31 03:20:25.041565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.041397 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" event={"ID":"bd3ab0f9-98e5-42b6-95fb-86317586e85c","Type":"ContainerStarted","Data":"6426476196f5c381ba13ac2f91f1b18df52ce3e99c55bf64a759221073ac987d"} Oct 31 03:20:25.097438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.096834 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podStartSLOduration=2.096812839 podStartE2EDuration="2.096812839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:25.090908205 +0000 UTC m=+1502.945925360" watchObservedRunningTime="2026-10-31 03:20:25.096812839 +0000 UTC m=+1502.951829983" Oct 31 03:20:25.126448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.124164 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:25.697439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.693798 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:20:25.709439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.704554 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:25.709439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.704643 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:25.917461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.916349 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:25.942440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.941811 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:25.946438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.945735 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:25.946438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:25.945819 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:26.181449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.180532 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:20:26.431382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.431336 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:20:26.433098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.433011 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:26.433282 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.433268 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8ee95d-54a2-42bd-a2eb-7e52cce8dd0b" containerName="istio-proxy" Oct 31 03:20:26.433434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.433422 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:26.433551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.433522 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e071bbc6-f4be-4a6c-a499-8eb27cc71926" containerName="istio-proxy" Oct 31 03:20:26.441284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.441169 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:20:26.441284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.441183 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.449636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.449607 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7269ae-openshift-default-dockercfg-kfww6\"" Oct 31 03:20:26.546675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.546633 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.546890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.546703 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.546890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.546851 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.546897 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs8p\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.547035 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.547094 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.547187 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.547327 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.547410 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.547397 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.648966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.648932 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.649288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.649262 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.649496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.649474 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs8p\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.649693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.649497 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.649793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.649681 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.649913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.649900 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650032 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650176 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650278 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650057 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650696 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.650919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.650879 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.651257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.651237 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.651542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.651516 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.652380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.652343 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.654344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.654215 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.662719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.662653 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs8p\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.663150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.663102 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.724724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.724617 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:26.725140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.725101 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:26.826097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.826062 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:26.864353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.864312 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:26.938441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.934867 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:26.938441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:26.934948 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:27.256062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.256002 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:20:27.256603 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:20:27.256516 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65a5e49_14c2_4907_b295_6ccc473195b5.slice/crio-ce369108ff60121c0aac1ba16a6c9858efcba40625da2e699a34c8da6455efbd WatchSource:0}: Error finding container ce369108ff60121c0aac1ba16a6c9858efcba40625da2e699a34c8da6455efbd: Status 404 returned error can't find the container with id ce369108ff60121c0aac1ba16a6c9858efcba40625da2e699a34c8da6455efbd Oct 31 03:20:27.259870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.259833 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:27.259992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.259920 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:27.259992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.259970 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:27.704443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.703800 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:27.704443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.703866 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:27.748437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.747568 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:27.935724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.935662 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:27.935918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.935731 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:27.941545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:27.941512 4556 scope.go:122] "RemoveContainer" containerID="2d7799deefe51ad072d445d5c667c40b310f5dca13156fddcdf441d36c962029" Oct 31 03:20:28.122442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.114602 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" event={"ID":"c65a5e49-14c2-4907-b295-6ccc473195b5","Type":"ContainerStarted","Data":"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8"} Oct 31 03:20:28.122442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.114701 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" event={"ID":"c65a5e49-14c2-4907-b295-6ccc473195b5","Type":"ContainerStarted","Data":"ce369108ff60121c0aac1ba16a6c9858efcba40625da2e699a34c8da6455efbd"} Oct 31 03:20:28.148447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.148092 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podStartSLOduration=2.148074141 podStartE2EDuration="2.148074141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:28.147141975 +0000 UTC m=+1506.002159154" watchObservedRunningTime="2026-10-31 03:20:28.148074141 +0000 UTC m=+1506.003091281" Oct 31 03:20:28.657637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.657543 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:28.700442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.692189 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.700442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.692252 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:28.830439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.827596 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:28.830439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.830356 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.830439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.830408 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:28.945503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.943954 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:28.945503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:28.944022 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:29.159197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.158846 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:20:29.608437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.606575 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:29.792076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.769790 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.792076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.769853 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:29.792076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.789518 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:29.869497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.840018 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.869497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.840079 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:29.939830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.939788 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:29.940105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.940076 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:29.968441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:29.960360 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:30.163122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.163080 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:20:30.221374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.221328 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 03:20:30.711441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.707799 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.711441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.707869 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:20:30.829440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.829156 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.829440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.829233 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:30.956355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.937050 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 31 03:20:30.958077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:30.958038 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 31 03:20:31.233440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.232637 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:31.704286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.704252 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:20:31.705676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.705647 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:20:31.827033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.826997 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:31.827204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.827053 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:31.939039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.939007 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:31.939563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.939546 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:31.940468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:31.940406 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:20:32.504017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:32.503980 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:32.827237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:32.827148 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:32.827237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:32.827220 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:33.529049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:33.529011 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:33.826540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:33.826459 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 31 03:20:33.826689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:33.826532 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 31 03:20:34.126366 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.126276 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:34.159525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.159469 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" podUID="6446de04-d3f8-4a36-ae3d-35a23618359a" containerName="payload-pre-processing" containerID="cri-o://8f675182544cce70118f92f29a2a177b29d80654efe2345a6bbb90a9220708c6" gracePeriod=25 Oct 31 03:20:34.340859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.340774 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:34.479488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.479392 4556 generic.go:356] "Generic (PLEG): container finished" podID="6446de04-d3f8-4a36-ae3d-35a23618359a" containerID="8f675182544cce70118f92f29a2a177b29d80654efe2345a6bbb90a9220708c6" exitCode=0 Oct 31 03:20:34.479907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.479512 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" event={"ID":"6446de04-d3f8-4a36-ae3d-35a23618359a","Type":"ContainerDied","Data":"8f675182544cce70118f92f29a2a177b29d80654efe2345a6bbb90a9220708c6"} Oct 31 03:20:34.569765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.569730 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:20:34.707206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.707112 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\" (UniqueName: \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\") pod \"6446de04-d3f8-4a36-ae3d-35a23618359a\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " Oct 31 03:20:34.707522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.707504 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\") pod \"6446de04-d3f8-4a36-ae3d-35a23618359a\" (UID: \"6446de04-d3f8-4a36-ae3d-35a23618359a\") " Oct 31 03:20:34.707877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.707850 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume" pod "6446de04-d3f8-4a36-ae3d-35a23618359a" (UID: "6446de04-d3f8-4a36-ae3d-35a23618359a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:34.708148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.708126 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6446de04-d3f8-4a36-ae3d-35a23618359a-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:34.709810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.709784 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k" pod "6446de04-d3f8-4a36-ae3d-35a23618359a" (UID: "6446de04-d3f8-4a36-ae3d-35a23618359a"). InnerVolumeSpecName "kube-api-access-kkx7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:34.810216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.810175 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkx7k\" (UniqueName: \"kubernetes.io/projected/6446de04-d3f8-4a36-ae3d-35a23618359a-kube-api-access-kkx7k\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:34.832255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.832221 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:34.833094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.832655 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:34.833562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:34.833536 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:20:35.206022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.205903 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:35.489611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.489516 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" Oct 31 03:20:35.489984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.489885 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc" event={"ID":"6446de04-d3f8-4a36-ae3d-35a23618359a","Type":"ContainerDied","Data":"00bd6b3700739846e5719e73569945149ee09e5f6a614d06c73c3c1e946e7b30"} Oct 31 03:20:35.490111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.490019 4556 scope.go:122] "RemoveContainer" containerID="8f675182544cce70118f92f29a2a177b29d80654efe2345a6bbb90a9220708c6" Oct 31 03:20:35.523356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.523321 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:20:35.528777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:35.528737 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3b8712-7b94557c79-bmxzc"] Oct 31 03:20:36.044032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:36.043998 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:36.194814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:36.194780 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:36.262976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:36.262939 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6446de04-d3f8-4a36-ae3d-35a23618359a" path="/var/lib/kubelet/pods/6446de04-d3f8-4a36-ae3d-35a23618359a/volumes" Oct 31 03:20:37.010714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:37.010680 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:37.144142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:37.144102 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:37.951942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:37.951910 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:38.844087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:38.844044 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:39.715252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:39.715213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:40.459258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:40.459214 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:41.150063 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:41.150026 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:41.969904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:41.969852 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:42.957839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:42.957765 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.655105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:43.655068 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:44.362300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:44.362253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:45.135403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:45.135368 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:45.953946 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:20:45.953902 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:20:45.969138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:45.969064 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:46.173916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:46.173876 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:46.947683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:46.947643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:47.736577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:47.736536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:48.630159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:48.630123 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:49.027784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:49.027738 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj"] Oct 31 03:20:49.031805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:49.031772 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b42ac26a-qr6lj"] Oct 31 03:20:49.583857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:49.583817 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:50.236097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:50.236038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:50.271304 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:50.271270 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27aacf07-8d5c-42ba-a035-1b7afdaba631" path="/var/lib/kubelet/pods/27aacf07-8d5c-42ba-a035-1b7afdaba631/volumes" Oct 31 03:20:50.974471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:50.974404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:51.934010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:51.933974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:52.070588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:52.070549 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:20:52.804018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:52.803924 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:53.850360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:53.850315 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:54.668009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:54.667971 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:55.612441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:55.612249 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.065378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.065335 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.513713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.513670 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:20:56.514552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.514529 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6446de04-d3f8-4a36-ae3d-35a23618359a" containerName="payload-pre-processing" Oct 31 03:20:56.514552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.514552 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6446de04-d3f8-4a36-ae3d-35a23618359a" containerName="payload-pre-processing" Oct 31 03:20:56.514768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.514725 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6446de04-d3f8-4a36-ae3d-35a23618359a" containerName="payload-pre-processing" Oct 31 03:20:56.521781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.521736 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.542817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.542779 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:20:56.605628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.605587 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.629660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629607 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629692 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629828 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629864 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629950 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.629981 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfqt\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.630077 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.630138 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.632001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.630278 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732275 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732331 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfqt\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732435 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732483 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732592 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732674 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732724 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732786 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732834 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732896 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.732965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.732938 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.733146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.733040 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.733340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.733318 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.733569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.733545 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.735306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.735274 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.735521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.735502 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.746143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.746109 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.746361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.746336 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfqt\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\") pod \"e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:56.940477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:56.940405 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:57.317244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.316475 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:20:57.317244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.316750 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" podUID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" containerName="maas-api" containerID="cri-o://d671ec234da92ce5872cf3087e4e253f228bc12d7672d3df1e703f9ceea04180" gracePeriod=30 Oct 31 03:20:57.329040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.325375 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:20:57.332305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.331535 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:57.332305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.331624 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:57.332305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.331669 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:20:57.399848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.399789 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:57.872814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.871348 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" event={"ID":"3e66071d-8b8e-4713-b328-5b86e2510379","Type":"ContainerStarted","Data":"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf"} Oct 31 03:20:57.872814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.871440 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" event={"ID":"3e66071d-8b8e-4713-b328-5b86e2510379","Type":"ContainerStarted","Data":"faddc6dd48348b400f5beb5cf1c98410c09d553f49bc6a8c56205f99ca45aaa7"} Oct 31 03:20:57.876625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.876591 4556 generic.go:356] "Generic (PLEG): container finished" podID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" containerID="d671ec234da92ce5872cf3087e4e253f228bc12d7672d3df1e703f9ceea04180" exitCode=0 Oct 31 03:20:57.877141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.876827 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" event={"ID":"e903a5bf-4b80-4f2a-8736-f0bd1a40792e","Type":"ContainerDied","Data":"d671ec234da92ce5872cf3087e4e253f228bc12d7672d3df1e703f9ceea04180"} Oct 31 03:20:57.922478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.917798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:57.942586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.942522 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:20:57.958743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:57.958708 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:20:58.019458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.015861 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podStartSLOduration=2.015835498 podStartE2EDuration="2.015835498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:57.918680026 +0000 UTC m=+1535.773697172" watchObservedRunningTime="2026-10-31 03:20:58.015835498 +0000 UTC m=+1535.870852640" Oct 31 03:20:58.047983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.047940 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\") pod \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " Oct 31 03:20:58.048148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.048038 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\" (UniqueName: \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\") pod \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " Oct 31 03:20:58.048190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.048182 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\") pod \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " Oct 31 03:20:58.048242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.048233 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\") pod \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\" (UID: \"e903a5bf-4b80-4f2a-8736-f0bd1a40792e\") " Oct 31 03:20:58.048393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.048354 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca" pod "e903a5bf-4b80-4f2a-8736-f0bd1a40792e" (UID: "e903a5bf-4b80-4f2a-8736-f0bd1a40792e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:20:58.049097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.049067 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.050865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.050823 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls" pod "e903a5bf-4b80-4f2a-8736-f0bd1a40792e" (UID: "e903a5bf-4b80-4f2a-8736-f0bd1a40792e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.051022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.050989 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs" pod "e903a5bf-4b80-4f2a-8736-f0bd1a40792e" (UID: "e903a5bf-4b80-4f2a-8736-f0bd1a40792e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:20:58.051164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.051147 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq" pod "e903a5bf-4b80-4f2a-8736-f0bd1a40792e" (UID: "e903a5bf-4b80-4f2a-8736-f0bd1a40792e"). InnerVolumeSpecName "kube-api-access-gmgmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:58.120786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.120746 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.155122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.155065 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgmq\" (UniqueName: \"kubernetes.io/projected/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-kube-api-access-gmgmq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.155309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.155127 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.155309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.155146 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e903a5bf-4b80-4f2a-8736-f0bd1a40792e-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:58.300271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.300231 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.627959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.627853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.901032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.900999 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" Oct 31 03:20:58.901032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.901018 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz" event={"ID":"e903a5bf-4b80-4f2a-8736-f0bd1a40792e","Type":"ContainerDied","Data":"1094fb1a66a6e9ef7a90a642f57d7bc5462d8dcbb7334be34ee1c46f64b20046"} Oct 31 03:20:58.901567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.901123 4556 scope.go:122] "RemoveContainer" containerID="d671ec234da92ce5872cf3087e4e253f228bc12d7672d3df1e703f9ceea04180" Oct 31 03:20:58.922496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.922459 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:20:58.929127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.929091 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3b8712-69954d7445-9b7kz"] Oct 31 03:20:58.942801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.942765 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:20:58.942979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:58.942838 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:20:59.121107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:59.121077 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:59.825936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:59.825885 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:59.941430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:59.941383 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:20:59.941833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:20:59.941471 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:00.261941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:00.261850 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" path="/var/lib/kubelet/pods/e903a5bf-4b80-4f2a-8736-f0bd1a40792e/volumes" Oct 31 03:21:00.705483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:00.705263 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:00.940955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:00.940908 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:21:00.941112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:00.940981 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:01.532467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:01.532427 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:01.823327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:01.823237 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:01.941179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:01.941143 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:21:01.941407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:01.941219 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:02.441080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:02.439409 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:02.798504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:02.789886 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:02.941231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:02.941178 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:21:02.941479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:02.941277 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:03.001797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:03.001704 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:03.510526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:03.510492 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:03.940958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:03.940916 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:21:03.941504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:03.940976 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:04.095054 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:04.094982 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:04.877038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:04.876948 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:04.941302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:04.941267 4556 patch_prober.go:29] interesting pod/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 31 03:21:04.941734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:04.941324 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 31 03:21:05.005580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.005539 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:21:05.005762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.005593 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:21:05.648862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.648749 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:05.971619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.971584 4556 generic.go:356] "Generic (PLEG): container finished" podID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerID="38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf" exitCode=0 Oct 31 03:21:05.972116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.971658 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerDied","Data":"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf"} Oct 31 03:21:05.972195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.972176 4556 scope.go:122] "RemoveContainer" containerID="38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf" Oct 31 03:21:05.974229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.974191 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:05.974359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.974291 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:05.974359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:05.974343 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:06.031169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:06.031140 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:21:06.032114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:06.032079 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:21:06.034011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:06.033985 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:21:06.402346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:06.402307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:06.997873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:06.997685 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerStarted","Data":"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d"} Oct 31 03:21:07.133817 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.133780 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:07.139687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.139658 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:21:07.139970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.139946 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:07.140029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.140011 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:07.313623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.311404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:07.850161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:07.850115 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:08.048032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.047996 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:08.140636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.140542 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:08.140798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.140627 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:08.345514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.345248 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:08.640218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.640178 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:21:08.641064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.641037 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" containerName="maas-api" Oct 31 03:21:08.641064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.641060 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" containerName="maas-api" Oct 31 03:21:08.641301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.641204 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e903a5bf-4b80-4f2a-8736-f0bd1a40792e" containerName="maas-api" Oct 31 03:21:08.645775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.645746 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.711203 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.711152 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:21:08.813442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.812997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:08.815812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.815782 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47c6j\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.815819 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816119 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816221 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816262 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816392 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816478 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816581 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.816978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.816657 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918457 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918579 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918698 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47c6j\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918727 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918964 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.918984 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919071 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919097 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919174 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919211 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919232 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919486 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.919750 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.921563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.920353 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.922832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.922810 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.932477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.932023 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.936883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.935726 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47c6j\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:08.936883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:08.936160 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\") pod \"e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:09.012339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.012050 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:09.101596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.101562 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:09.140317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.140276 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:09.140571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.140347 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:09.341161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.341129 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:21:09.343292 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:21:09.343260 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a20ec2_690a_4360_9287_a2f8f85f09b4.slice/crio-8d2d8785a392de23b46d33b6f12f4bab3ce31dea2a1fb5e5e7a82217ed374e0e WatchSource:0}: Error finding container 8d2d8785a392de23b46d33b6f12f4bab3ce31dea2a1fb5e5e7a82217ed374e0e: Status 404 returned error can't find the container with id 8d2d8785a392de23b46d33b6f12f4bab3ce31dea2a1fb5e5e7a82217ed374e0e Oct 31 03:21:09.346328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.346293 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:09.346454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.346369 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:09.346454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.346402 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:21:09.360658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:09.360625 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:10.050708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.050668 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" event={"ID":"75a20ec2-690a-4360-9287-a2f8f85f09b4","Type":"ContainerStarted","Data":"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6"} Oct 31 03:21:10.050931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.050739 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" event={"ID":"75a20ec2-690a-4360-9287-a2f8f85f09b4","Type":"ContainerStarted","Data":"8d2d8785a392de23b46d33b6f12f4bab3ce31dea2a1fb5e5e7a82217ed374e0e"} Oct 31 03:21:10.090440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.088845 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podStartSLOduration=2.088823499 podStartE2EDuration="2.088823499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.086947138 +0000 UTC m=+1547.941964299" watchObservedRunningTime="2026-10-31 03:21:10.088823499 +0000 UTC m=+1547.943840621" Oct 31 03:21:10.135439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.135344 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:10.151442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.144818 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:10.151442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:10.144881 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:11.016438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.015393 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:11.016438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.015610 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 03:21:11.016438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.015682 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 03:21:11.064178 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.064139 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:11.140237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.140195 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:11.140814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.140252 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:11.876935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:11.874739 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.017442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:12.016894 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 03:21:12.017442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:12.017004 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 03:21:12.140478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:12.140343 4556 patch_prober.go:29] interesting pod/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 31 03:21:12.140478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:12.140448 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 31 03:21:12.899439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:12.895875 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:13.013375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.013342 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 03:21:13.013661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.013399 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 03:21:13.144764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.144731 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:21:13.145249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.145214 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:21:13.145954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.145935 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:21:13.698716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.698680 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:21:13.706511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.706402 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:21:13.712912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.712875 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:21:13.923795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:13.923753 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:14.013887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:14.013801 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 03:21:14.013887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:14.013855 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 03:21:14.135495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:14.135463 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:14.900827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:14.891303 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:15.013604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:15.013571 4556 patch_prober.go:29] interesting pod/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 03:21:15.013792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:15.013628 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 03:21:15.940904 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:15.940858 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:21:16.081540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:16.078917 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:16.132702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:16.132660 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:16.135729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:16.134980 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:16.138164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:16.138134 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:21:16.927022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:16.926983 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:17.665409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:17.665369 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:18.489769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:18.489730 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:18.991568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:18.991533 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:21:18.998627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:18.998594 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:21:19.128140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:19.128071 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:19.350640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:19.350110 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:20.070476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:20.066706 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:20.783823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:20.783777 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:21.694510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:21.694474 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:22.527901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:22.527838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:23.476597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:23.476561 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:23.992140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:23.992090 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" containerID="cri-o://d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72" gracePeriod=25 Oct 31 03:21:23.999872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:23.999824 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" containerID="cri-o://c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5" gracePeriod=25 Oct 31 03:21:24.194387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:24.194348 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:24.639988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:24.639948 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:25.159974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:25.159939 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:26.013765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:26.013711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:26.967548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:26.967508 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:27.689561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:27.689515 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:28.500807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:28.500718 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:28.731509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:28.731467 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" containerID="cri-o://04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a" gracePeriod=45 Oct 31 03:21:28.735322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:28.735285 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" containerID="cri-o://f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476" gracePeriod=45 Oct 31 03:21:28.746000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:28.745953 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" containerID="cri-o://d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a" gracePeriod=45 Oct 31 03:21:28.847722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:28.847672 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Oct 31 03:21:29.065862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.065829 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:21:29.071228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.071200 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:21:29.121726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.121691 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:21:29.162646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162605 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.162858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162671 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.162858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162704 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.162858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162777 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.162858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162819 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.163114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.162954 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.163114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163010 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\" (UniqueName: \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.163114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163041 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.163114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163112 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.163338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163147 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\" (UniqueName: \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.163338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163158 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.163338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163242 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.163338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163292 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\" (UniqueName: \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\") pod \"914c3f84-6dbe-4652-a97d-6125aa9db86d\" (UID: \"914c3f84-6dbe-4652-a97d-6125aa9db86d\") " Oct 31 03:21:29.163338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.163328 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\") pod \"59be27b7-2cc1-4119-8c9c-4964760df2a7\" (UID: \"59be27b7-2cc1-4119-8c9c-4964760df2a7\") " Oct 31 03:21:29.164334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.164304 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.164509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.164362 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.165056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.164990 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.165056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.165022 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.165231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.165218 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.165476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.165451 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.170909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.170867 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.175042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.174795 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:29.175042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.174892 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:29.175042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.174999 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "kube-api-access-2bfbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:29.177514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.177376 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm" pod "59be27b7-2cc1-4119-8c9c-4964760df2a7" (UID: "59be27b7-2cc1-4119-8c9c-4964760df2a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.177909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.177888 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb" pod "914c3f84-6dbe-4652-a97d-6125aa9db86d" (UID: "914c3f84-6dbe-4652-a97d-6125aa9db86d"). InnerVolumeSpecName "kube-api-access-lfnkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:29.271233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.260369 4556 generic.go:356] "Generic (PLEG): container finished" podID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerID="f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476" exitCode=0 Oct 31 03:21:29.271233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.260434 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" Oct 31 03:21:29.271233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.260542 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" event={"ID":"914c3f84-6dbe-4652-a97d-6125aa9db86d","Type":"ContainerDied","Data":"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476"} Oct 31 03:21:29.271233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.260598 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx" event={"ID":"914c3f84-6dbe-4652-a97d-6125aa9db86d","Type":"ContainerDied","Data":"fb2ddaf66dacbc5e8c06ef5c1b8c0c95672e7ab1a179989797b2eb7b2658880b"} Oct 31 03:21:29.271233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.260623 4556 scope.go:122] "RemoveContainer" containerID="f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476" Oct 31 03:21:29.277484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277308 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.277484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277392 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.277759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277494 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\" (UniqueName: \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.277759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277553 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.277759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277607 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.277932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.277791 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\") pod \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\" (UID: \"69f434bd-237c-4c9f-aa3d-af3599c7feeb\") " Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278408 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfbz\" (UniqueName: \"kubernetes.io/projected/59be27b7-2cc1-4119-8c9c-4964760df2a7-kube-api-access-2bfbz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278449 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59be27b7-2cc1-4119-8c9c-4964760df2a7-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278463 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278474 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/914c3f84-6dbe-4652-a97d-6125aa9db86d-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278484 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278528 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfnkb\" (UniqueName: \"kubernetes.io/projected/914c3f84-6dbe-4652-a97d-6125aa9db86d-kube-api-access-lfnkb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278545 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278559 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59be27b7-2cc1-4119-8c9c-4964760df2a7-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278572 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.278588 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/914c3f84-6dbe-4652-a97d-6125aa9db86d-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.282503 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.283290 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.284843 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.285057 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.286025 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "kube-api-access-spshc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:29.288522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.286745 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:29.289877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.289704 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm" pod "69f434bd-237c-4c9f-aa3d-af3599c7feeb" (UID: "69f434bd-237c-4c9f-aa3d-af3599c7feeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:29.289877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.289727 4556 generic.go:356] "Generic (PLEG): container finished" podID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerID="04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a" exitCode=0 Oct 31 03:21:29.289877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.289776 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" Oct 31 03:21:29.289877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.289813 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" event={"ID":"59be27b7-2cc1-4119-8c9c-4964760df2a7","Type":"ContainerDied","Data":"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a"} Oct 31 03:21:29.289877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.289872 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz" event={"ID":"59be27b7-2cc1-4119-8c9c-4964760df2a7","Type":"ContainerDied","Data":"7ea85a757d4d4fb12a5db5d5caa8f8dce6d9cd133e9450ef891d069d34c5b687"} Oct 31 03:21:29.293176 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.293140 4556 generic.go:356] "Generic (PLEG): container finished" podID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerID="d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a" exitCode=0 Oct 31 03:21:29.293368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.293237 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" event={"ID":"69f434bd-237c-4c9f-aa3d-af3599c7feeb","Type":"ContainerDied","Data":"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a"} Oct 31 03:21:29.293368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.293270 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" event={"ID":"69f434bd-237c-4c9f-aa3d-af3599c7feeb","Type":"ContainerDied","Data":"07628ae33697e0aa1cd0d337acc6d98042dfdb71ff8113cc80208d2ac3f419d7"} Oct 31 03:21:29.293368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.293301 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg" Oct 31 03:21:29.303772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.303731 4556 scope.go:122] "RemoveContainer" containerID="f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476" Oct 31 03:21:29.304230 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:29.304198 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476\": container with ID starting with f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476 not found: ID does not exist" containerID="f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476" Oct 31 03:21:29.304344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.304244 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476"} err="failed to get container status \"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476\": rpc error: code = NotFound desc = could not find container \"f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476\": container with ID starting with f6d172bee392aa12e94a1f7e0fdad87a53c1fc6787843d5cd52e1202c3979476 not found: ID does not exist" Oct 31 03:21:29.304344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.304271 4556 scope.go:122] "RemoveContainer" containerID="04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a" Oct 31 03:21:29.314196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.314162 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:21:29.322769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.322732 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-w3-99bc97-kserve-7d6cf64759-525jx"] Oct 31 03:21:29.323730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.323676 4556 scope.go:122] "RemoveContainer" containerID="04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a" Oct 31 03:21:29.324221 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:29.324190 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a\": container with ID starting with 04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a not found: ID does not exist" containerID="04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a" Oct 31 03:21:29.324336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.324230 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a"} err="failed to get container status \"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a\": rpc error: code = NotFound desc = could not find container \"04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a\": container with ID starting with 04a14b59924f29658faa9bd1d80784cabb3f724c02066908bfe7e7903cd5a80a not found: ID does not exist" Oct 31 03:21:29.324336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.324287 4556 scope.go:122] "RemoveContainer" containerID="d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a" Oct 31 03:21:29.336989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.336946 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:21:29.343045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.342991 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/simulator-w3-99bc97-kserve-79f978b49-d62mg"] Oct 31 03:21:29.349815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.349784 4556 scope.go:122] "RemoveContainer" containerID="d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a" Oct 31 03:21:29.350212 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:29.350180 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a\": container with ID starting with d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a not found: ID does not exist" containerID="d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a" Oct 31 03:21:29.350350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.350225 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a"} err="failed to get container status \"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a\": rpc error: code = NotFound desc = could not find container \"d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a\": container with ID starting with d4100de34d76933f14b0994d48dfc715e4cd6a44e6681a5197da52fad937f08a not found: ID does not exist" Oct 31 03:21:29.354089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.354061 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:21:29.357894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.357864 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-99bc97/distinct-2-w3-99bc97-kserve-b6f848f68-vgtbz"] Oct 31 03:21:29.380771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380730 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.380771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380766 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.380771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380778 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spshc\" (UniqueName: \"kubernetes.io/projected/69f434bd-237c-4c9f-aa3d-af3599c7feeb-kube-api-access-spshc\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.381029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380791 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.381029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380800 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f434bd-237c-4c9f-aa3d-af3599c7feeb-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:29.381029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:29.380808 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f434bd-237c-4c9f-aa3d-af3599c7feeb-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:30.099359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:30.099205 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:30.265537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:30.265488 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" path="/var/lib/kubelet/pods/59be27b7-2cc1-4119-8c9c-4964760df2a7/volumes" Oct 31 03:21:30.266982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:30.266946 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" path="/var/lib/kubelet/pods/69f434bd-237c-4c9f-aa3d-af3599c7feeb/volumes" Oct 31 03:21:30.267940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:30.267908 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" path="/var/lib/kubelet/pods/914c3f84-6dbe-4652-a97d-6125aa9db86d/volumes" Oct 31 03:21:30.998359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:30.998326 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:31.900374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:31.900332 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:21:31.900845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:31.900606 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:21:32.006900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:32.006847 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:32.234318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:32.234218 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:21:33.084740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:33.084686 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:34.161816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:34.161720 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:35.393446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:35.392324 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:36.497439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:36.482201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:36.756264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:36.756172 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:37.200736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:37.200697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:37.748001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:37.747958 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:38.562304 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:38.562263 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:39.061045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.060999 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:21:39.061484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.061249 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" podUID="e014c245-4973-4be6-9968-2bb75c5fda94" containerName="maas-api" containerID="cri-o://918518a4ea6e6be774790ff32748bdffa35e717b08f7e9512b820022cdd37da6" gracePeriod=30 Oct 31 03:21:39.226342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.226305 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:39.541603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.534379 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:39.586978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.586102 4556 generic.go:356] "Generic (PLEG): container finished" podID="e014c245-4973-4be6-9968-2bb75c5fda94" containerID="918518a4ea6e6be774790ff32748bdffa35e717b08f7e9512b820022cdd37da6" exitCode=0 Oct 31 03:21:39.586978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.586306 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" event={"ID":"e014c245-4973-4be6-9968-2bb75c5fda94","Type":"ContainerDied","Data":"918518a4ea6e6be774790ff32748bdffa35e717b08f7e9512b820022cdd37da6"} Oct 31 03:21:39.610163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.610128 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:21:39.647603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.647561 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\") pod \"e014c245-4973-4be6-9968-2bb75c5fda94\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " Oct 31 03:21:39.648186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.648155 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\") pod \"e014c245-4973-4be6-9968-2bb75c5fda94\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " Oct 31 03:21:39.648400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.648382 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\") pod \"e014c245-4973-4be6-9968-2bb75c5fda94\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " Oct 31 03:21:39.648935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.648573 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\" (UniqueName: \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\") pod \"e014c245-4973-4be6-9968-2bb75c5fda94\" (UID: \"e014c245-4973-4be6-9968-2bb75c5fda94\") " Oct 31 03:21:39.649302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.649279 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca" pod "e014c245-4973-4be6-9968-2bb75c5fda94" (UID: "e014c245-4973-4be6-9968-2bb75c5fda94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:39.650716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.650683 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e014c245-4973-4be6-9968-2bb75c5fda94-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:39.656132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.656088 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls" pod "e014c245-4973-4be6-9968-2bb75c5fda94" (UID: "e014c245-4973-4be6-9968-2bb75c5fda94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:39.659154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.659112 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs" pod "e014c245-4973-4be6-9968-2bb75c5fda94" (UID: "e014c245-4973-4be6-9968-2bb75c5fda94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:21:39.659368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.659348 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw" pod "e014c245-4973-4be6-9968-2bb75c5fda94" (UID: "e014c245-4973-4be6-9968-2bb75c5fda94"). InnerVolumeSpecName "kube-api-access-9gtdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:39.753811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.753771 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:39.753811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.753812 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gtdw\" (UniqueName: \"kubernetes.io/projected/e014c245-4973-4be6-9968-2bb75c5fda94-kube-api-access-9gtdw\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:39.754196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.753829 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e014c245-4973-4be6-9968-2bb75c5fda94-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:39.765588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:39.765545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:40.069876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.069837 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:40.601488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.601455 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" Oct 31 03:21:40.601709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.601409 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch" event={"ID":"e014c245-4973-4be6-9968-2bb75c5fda94","Type":"ContainerDied","Data":"a74d50e1c01654d573cd826d266db5cdc3d94888aa787892c0f56d0b6280ff9c"} Oct 31 03:21:40.601709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.601607 4556 scope.go:122] "RemoveContainer" containerID="918518a4ea6e6be774790ff32748bdffa35e717b08f7e9512b820022cdd37da6" Oct 31 03:21:40.627140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.627100 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:21:40.631117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.631087 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-fd2e0b-869fff6f5-rf7ch"] Oct 31 03:21:40.667508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:40.667472 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:41.317064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:41.317025 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:41.601750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:41.601646 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:41.778016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:41.777975 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:41.901271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:41.901231 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:21:41.901660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:41.901636 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:21:42.259972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:42.259892 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e014c245-4973-4be6-9968-2bb75c5fda94" path="/var/lib/kubelet/pods/e014c245-4973-4be6-9968-2bb75c5fda94/volumes" Oct 31 03:21:42.347708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:42.347665 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:42.867846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:42.867804 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:43.739571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:43.739378 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:44.313878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.313844 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:21:44.355571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.355537 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:21:44.442088 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.442056 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\") pod \"314bff69-cff9-4977-a24c-d9b7d65f0a40\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " Oct 31 03:21:44.442272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.442154 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\" (UniqueName: \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\") pod \"314bff69-cff9-4977-a24c-d9b7d65f0a40\" (UID: \"314bff69-cff9-4977-a24c-d9b7d65f0a40\") " Oct 31 03:21:44.442321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.442294 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\" (UniqueName: \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\") pod \"97f9780b-1631-4c9f-bedd-df67d21f1d83\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " Oct 31 03:21:44.442577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.442547 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume" pod "314bff69-cff9-4977-a24c-d9b7d65f0a40" (UID: "314bff69-cff9-4977-a24c-d9b7d65f0a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:44.442577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.442569 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\") pod \"97f9780b-1631-4c9f-bedd-df67d21f1d83\" (UID: \"97f9780b-1631-4c9f-bedd-df67d21f1d83\") " Oct 31 03:21:44.443029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.443004 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume" pod "97f9780b-1631-4c9f-bedd-df67d21f1d83" (UID: "97f9780b-1631-4c9f-bedd-df67d21f1d83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:44.443285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.443263 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f9780b-1631-4c9f-bedd-df67d21f1d83-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:44.443352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.443294 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314bff69-cff9-4977-a24c-d9b7d65f0a40-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:44.444570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.444547 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj" pod "314bff69-cff9-4977-a24c-d9b7d65f0a40" (UID: "314bff69-cff9-4977-a24c-d9b7d65f0a40"). InnerVolumeSpecName "kube-api-access-sqsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:44.444703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.444581 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn" pod "97f9780b-1631-4c9f-bedd-df67d21f1d83" (UID: "97f9780b-1631-4c9f-bedd-df67d21f1d83"). InnerVolumeSpecName "kube-api-access-ww6fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:44.546966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.546852 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqsvj\" (UniqueName: \"kubernetes.io/projected/314bff69-cff9-4977-a24c-d9b7d65f0a40-kube-api-access-sqsvj\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:44.567535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.567506 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6fn\" (UniqueName: \"kubernetes.io/projected/97f9780b-1631-4c9f-bedd-df67d21f1d83-kube-api-access-ww6fn\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:44.567789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.567276 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:44.689229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.689189 4556 generic.go:356] "Generic (PLEG): container finished" podID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerID="d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72" exitCode=0 Oct 31 03:21:44.690876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.689607 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" Oct 31 03:21:44.691316 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.689485 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" event={"ID":"314bff69-cff9-4977-a24c-d9b7d65f0a40","Type":"ContainerDied","Data":"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72"} Oct 31 03:21:44.691555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.691527 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v" event={"ID":"314bff69-cff9-4977-a24c-d9b7d65f0a40","Type":"ContainerDied","Data":"e003d5ddca362cd37d267c85fc100c627bc9f5c44f34935790861c467547d9c8"} Oct 31 03:21:44.691671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.691660 4556 scope.go:122] "RemoveContainer" containerID="d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72" Oct 31 03:21:44.694848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.694825 4556 generic.go:356] "Generic (PLEG): container finished" podID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerID="c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5" exitCode=0 Oct 31 03:21:44.695102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.695088 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" event={"ID":"97f9780b-1631-4c9f-bedd-df67d21f1d83","Type":"ContainerDied","Data":"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5"} Oct 31 03:21:44.695213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.695201 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" event={"ID":"97f9780b-1631-4c9f-bedd-df67d21f1d83","Type":"ContainerDied","Data":"2b7c886f020f4190b837bf167c134f1b557d7e97b0711c585f33a6981abfd6e7"} Oct 31 03:21:44.695340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.695329 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql" Oct 31 03:21:44.709044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.709022 4556 scope.go:122] "RemoveContainer" containerID="d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72" Oct 31 03:21:44.709561 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:44.709540 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72\": container with ID starting with d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72 not found: ID does not exist" containerID="d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72" Oct 31 03:21:44.709675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.709567 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72"} err="failed to get container status \"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72\": rpc error: code = NotFound desc = could not find container \"d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72\": container with ID starting with d99144969e9e9107cb68851a19d163f7530b30a04a00dd7dbe4c8528b97b5a72 not found: ID does not exist" Oct 31 03:21:44.709675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.709582 4556 scope.go:122] "RemoveContainer" containerID="c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5" Oct 31 03:21:44.726530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.726482 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:21:44.727913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.727791 4556 scope.go:122] "RemoveContainer" containerID="c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5" Oct 31 03:21:44.728371 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:44.728328 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5\": container with ID starting with c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5 not found: ID does not exist" containerID="c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5" Oct 31 03:21:44.728587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.728372 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5"} err="failed to get container status \"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5\": rpc error: code = NotFound desc = could not find container \"c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5\": container with ID starting with c65962f58975f39681b8b7a99d265687a23a0b0f437afe90e1b65005b8fdbfe5 not found: ID does not exist" Oct 31 03:21:44.731656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.731624 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-99bc97-6f76d77d99-zbp4v"] Oct 31 03:21:44.736198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.736165 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:21:44.744998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:44.744958 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-99bc97-6c88d4786c-r6qql"] Oct 31 03:21:45.487775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:45.487732 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:45.947576 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:45.947537 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:21:46.022772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.022728 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:46.262461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.262359 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" path="/var/lib/kubelet/pods/314bff69-cff9-4977-a24c-d9b7d65f0a40/volumes" Oct 31 03:21:46.262981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.262963 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" path="/var/lib/kubelet/pods/97f9780b-1631-4c9f-bedd-df67d21f1d83/volumes" Oct 31 03:21:46.264571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.264536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:46.475435 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.474974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:46.917560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:46.915946 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:48.013359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:48.013325 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:48.896268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:48.896236 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:49.178887 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:49.178855 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:49.610800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:49.610717 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:49.927552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:49.927517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:50.799575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:50.799542 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:51.876441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:51.876367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:52.040408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:52.040378 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:52.268703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:52.268664 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:52.562497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:52.561893 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:52.796957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:52.796907 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:53.117440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:53.112719 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:53.548195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:53.548157 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:53.767230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:53.767183 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:21:53.767618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:53.767584 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" containerID="cri-o://377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58" gracePeriod=30 Oct 31 03:21:53.776594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:53.776560 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:54.057317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:54.057277 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:54.292395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:54.292345 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:54.958138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:54.958096 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:55.869712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:55.869677 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:56.045847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:56.045812 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:56.846130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:56.846090 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:57.139954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:57.139869 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:57.329209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:57.329171 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:57.738687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:57.738637 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:58.633891 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:58.633850 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:59.088979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.088951 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:21:59.219691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.219591 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.219691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.219686 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.219953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.219737 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220006 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220045 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220091 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220122 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.220240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220155 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220163 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:21:59.220393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220261 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.220393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.220301 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\") pod \"198e0fdd-69de-4242-83f7-bee12900dc7c\" (UID: \"198e0fdd-69de-4242-83f7-bee12900dc7c\") " Oct 31 03:21:59.221078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.221046 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198e0fdd-69de-4242-83f7-bee12900dc7c-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.221195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.221081 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.221587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.221517 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.221830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.221758 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.222671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.222641 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:21:59.222766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.222740 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.224646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.224609 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "kube-api-access-xh426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.224646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.224635 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:21:59.224782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.224609 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy" pod "198e0fdd-69de-4242-83f7-bee12900dc7c" (UID: "198e0fdd-69de-4242-83f7-bee12900dc7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:21:59.323270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323224 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323264 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323281 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323298 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh426\" (UniqueName: \"kubernetes.io/projected/198e0fdd-69de-4242-83f7-bee12900dc7c-kube-api-access-xh426\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323316 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323330 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198e0fdd-69de-4242-83f7-bee12900dc7c-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.323521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.323343 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198e0fdd-69de-4242-83f7-bee12900dc7c-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:21:59.553812 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.553690 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:59.843024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.842941 4556 generic.go:356] "Generic (PLEG): container finished" podID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerID="377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58" exitCode=0 Oct 31 03:21:59.843024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.842988 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" Oct 31 03:21:59.843526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.843029 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerDied","Data":"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58"} Oct 31 03:21:59.843526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.843086 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr" event={"ID":"198e0fdd-69de-4242-83f7-bee12900dc7c","Type":"ContainerDied","Data":"666aa2a7cee6b6621e666f06ce18beb1198878f8fdbcdf69bc037924854f9045"} Oct 31 03:21:59.843526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.843116 4556 scope.go:122] "RemoveContainer" containerID="377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58" Oct 31 03:21:59.855913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.855888 4556 scope.go:122] "RemoveContainer" containerID="1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347" Oct 31 03:21:59.868223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.868199 4556 scope.go:122] "RemoveContainer" containerID="377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58" Oct 31 03:21:59.868935 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:59.868824 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58\": container with ID starting with 377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58 not found: ID does not exist" containerID="377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58" Oct 31 03:21:59.868935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.868868 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58"} err="failed to get container status \"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58\": rpc error: code = NotFound desc = could not find container \"377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58\": container with ID starting with 377e82c1974076435eef82138a1da26c49f188c3d5a8b11373fa9986073e8d58 not found: ID does not exist" Oct 31 03:21:59.868935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.868894 4556 scope.go:122] "RemoveContainer" containerID="1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347" Oct 31 03:21:59.869230 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:21:59.869210 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347\": container with ID starting with 1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347 not found: ID does not exist" containerID="1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347" Oct 31 03:21:59.869293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.869233 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347"} err="failed to get container status \"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347\": rpc error: code = NotFound desc = could not find container \"1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347\": container with ID starting with 1127d81a241fef6d9840f0ff1df2e8acc4327e29b6709c4e7006906c31b80347 not found: ID does not exist" Oct 31 03:21:59.869293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.869285 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:21:59.873601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.873566 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-99bc97-openshift-default-7f9f6fcb45-twcvr"] Oct 31 03:21:59.913078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:21:59.913041 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:22:00.023774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:00.023736 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:00.210451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:00.209830 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:00.321552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:00.319033 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" path="/var/lib/kubelet/pods/198e0fdd-69de-4242-83f7-bee12900dc7c/volumes" Oct 31 03:22:00.465886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:00.465748 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:01.180439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:01.169039 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:01.952107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:01.952058 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:02.797165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:02.797123 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:03.552286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:03.552178 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:04.018603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:04.018554 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:04.363500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:04.362926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:04.557094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:04.557039 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:04.915436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:04.913564 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" podUID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" containerName="payload-pre-processing" containerID="cri-o://f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158" gracePeriod=25 Oct 31 03:22:05.065512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.045853 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:05.319667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.319632 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:22:05.421923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.421888 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\" (UniqueName: \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\") pod \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " Oct 31 03:22:05.422384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.422356 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\") pod \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\" (UID: \"c0e6036e-25e1-4103-ba8b-4d3f4048db2a\") " Oct 31 03:22:05.423623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.422911 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume" pod "c0e6036e-25e1-4103-ba8b-4d3f4048db2a" (UID: "c0e6036e-25e1-4103-ba8b-4d3f4048db2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:05.426398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.425556 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:05.430883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.430848 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746" pod "c0e6036e-25e1-4103-ba8b-4d3f4048db2a" (UID: "c0e6036e-25e1-4103-ba8b-4d3f4048db2a"). InnerVolumeSpecName "kube-api-access-lq746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:05.528663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.528571 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq746\" (UniqueName: \"kubernetes.io/projected/c0e6036e-25e1-4103-ba8b-4d3f4048db2a-kube-api-access-lq746\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:05.637660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.637613 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:05.933498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.933460 4556 generic.go:356] "Generic (PLEG): container finished" podID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" containerID="f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158" exitCode=0 Oct 31 03:22:05.933683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.933508 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" Oct 31 03:22:05.933683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.933556 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" event={"ID":"c0e6036e-25e1-4103-ba8b-4d3f4048db2a","Type":"ContainerDied","Data":"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158"} Oct 31 03:22:05.933683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.933619 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59" event={"ID":"c0e6036e-25e1-4103-ba8b-4d3f4048db2a","Type":"ContainerDied","Data":"63bd08cd31dcb4acd9b4ee743c3a68e498d0bdb5555a5f9138861a8a1eb344e8"} Oct 31 03:22:05.933683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.933651 4556 scope.go:122] "RemoveContainer" containerID="f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158" Oct 31 03:22:05.948676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.948643 4556 scope.go:122] "RemoveContainer" containerID="f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158" Oct 31 03:22:05.949008 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:05.948988 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158\": container with ID starting with f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158 not found: ID does not exist" containerID="f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158" Oct 31 03:22:05.949084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.949016 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158"} err="failed to get container status \"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158\": rpc error: code = NotFound desc = could not find container \"f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158\": container with ID starting with f5fbdbd534096cd91142424ef690f207e2cf6e69511feda63d38388b4336f158 not found: ID does not exist" Oct 31 03:22:05.963096 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.963060 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:22:05.993886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:05.993842 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-677281-6b745f56dc-bdr59"] Oct 31 03:22:06.260109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:06.260027 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" path="/var/lib/kubelet/pods/c0e6036e-25e1-4103-ba8b-4d3f4048db2a/volumes" Oct 31 03:22:06.379821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:06.379782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:07.164095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:07.164047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:07.963221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:07.963184 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:08.631990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:08.631942 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:09.402003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:09.401961 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:09.601385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:09.601349 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:10.242615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.236974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:10.418317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.418272 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:10.866551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.866512 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": read tcp 10.134.0.2:36280->10.134.0.148:15021: read: connection reset by peer" start-of-body= Oct 31 03:22:10.866726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.866611 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": read tcp 10.134.0.2:36280->10.134.0.148:15021: read: connection reset by peer" Oct 31 03:22:10.977198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.977166 4556 generic.go:356] "Generic (PLEG): container finished" podID="263b74e5-a37c-4823-8232-317ea36efd3e" containerID="0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46" exitCode=0 Oct 31 03:22:10.977391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.977247 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerDied","Data":"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46"} Oct 31 03:22:10.977884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.977866 4556 scope.go:122] "RemoveContainer" containerID="0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46" Oct 31 03:22:10.979953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.979925 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:22:10.980042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.979997 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:22:10.980082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:10.980041 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:22:11.267714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:11.267667 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:11.992438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:11.992336 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerStarted","Data":"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a"} Oct 31 03:22:12.088441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:12.082793 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:12.196594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:12.196548 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:22:12.196813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:12.196743 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:22:12.196885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:12.196816 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 03:22:12.859368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:12.859333 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:13.196145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:13.196109 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:22:13.196579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:13.196163 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 03:22:13.757546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:13.757506 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:14.196218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.196183 4556 patch_prober.go:29] interesting pod/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:22:14.196763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.196245 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 31 03:22:14.334505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.334471 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:14.513758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.513676 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:14.625506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.625472 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:22:14.647541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.647490 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" containerID="cri-o://caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a" gracePeriod=30 Oct 31 03:22:14.754498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:14.750397 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:15.208087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:15.208047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:15.798591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:15.798536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:15.941995 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:15.941922 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:22:16.833440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:16.831205 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:17.661022 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:17.660985 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:18.335505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:18.335468 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:19.117459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:19.117404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:20.018464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:20.018373 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:20.127252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:20.127208 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:22:20.302124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:20.302008 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:20.815054 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:20.815015 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:21.683734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:21.683685 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:22.234056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.234010 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:22.423711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.423665 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:22.622893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.622857 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:22.805017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.804973 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:22.901660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.901623 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:22:22.964483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964403 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964535 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964571 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964626 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964699 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964787 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.964872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964866 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.965048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.964899 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.965048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965019 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\") pod \"263b74e5-a37c-4823-8232-317ea36efd3e\" (UID: \"263b74e5-a37c-4823-8232-317ea36efd3e\") " Oct 31 03:22:22.965164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965092 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:22.965164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965096 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.965677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965469 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.965803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965696 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.966182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.965888 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.966294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.966218 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263b74e5-a37c-4823-8232-317ea36efd3e-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.966294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.966243 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.966294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.966258 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.966294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.966272 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.966294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.966285 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:22.967911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.967882 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:22.968210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.968086 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:22:22.968364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.968343 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "kube-api-access-jrhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:22.968767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:22.968751 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token" pod "263b74e5-a37c-4823-8232-317ea36efd3e" (UID: "263b74e5-a37c-4823-8232-317ea36efd3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:23.068277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.068232 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263b74e5-a37c-4823-8232-317ea36efd3e-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:23.068277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.068269 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhbs\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-kube-api-access-jrhbs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:23.068277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.068281 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263b74e5-a37c-4823-8232-317ea36efd3e-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:23.068585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.068295 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263b74e5-a37c-4823-8232-317ea36efd3e-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:23.109100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.109066 4556 generic.go:356] "Generic (PLEG): container finished" podID="263b74e5-a37c-4823-8232-317ea36efd3e" containerID="caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a" exitCode=0 Oct 31 03:22:23.109250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.109117 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" Oct 31 03:22:23.109250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.109154 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerDied","Data":"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a"} Oct 31 03:22:23.109250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.109211 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl" event={"ID":"263b74e5-a37c-4823-8232-317ea36efd3e","Type":"ContainerDied","Data":"9a5c5d0ce2544b0b8aea433f26f47e5d0f1009ecab0c013d5bab0ac872e5638e"} Oct 31 03:22:23.109497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.109236 4556 scope.go:122] "RemoveContainer" containerID="caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a" Oct 31 03:22:23.121262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.121235 4556 scope.go:122] "RemoveContainer" containerID="0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46" Oct 31 03:22:23.134906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.133095 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:22:23.134906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.134036 4556 scope.go:122] "RemoveContainer" containerID="caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a" Oct 31 03:22:23.137335 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:23.137294 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a\": container with ID starting with caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a not found: ID does not exist" containerID="caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a" Oct 31 03:22:23.137514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.137347 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a"} err="failed to get container status \"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a\": rpc error: code = NotFound desc = could not find container \"caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a\": container with ID starting with caae19ac97ac591b6689760d5fc5f942484b582a3804d52d10129dca3ea46c4a not found: ID does not exist" Oct 31 03:22:23.137514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.137374 4556 scope.go:122] "RemoveContainer" containerID="0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46" Oct 31 03:22:23.138007 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:23.137977 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46\": container with ID starting with 0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46 not found: ID does not exist" containerID="0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46" Oct 31 03:22:23.138059 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.138017 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46"} err="failed to get container status \"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46\": rpc error: code = NotFound desc = could not find container \"0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46\": container with ID starting with 0d67383a536307ce05c874379c47847bf99e350cb53757abad29105331996e46 not found: ID does not exist" Oct 31 03:22:23.140263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.140238 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-677281-openshift-default-75fcf58899-5vjfl"] Oct 31 03:22:23.168075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.168042 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:23.480638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:23.480601 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:24.228101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:24.227988 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:24.260659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:24.260621 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" path="/var/lib/kubelet/pods/263b74e5-a37c-4823-8232-317ea36efd3e/volumes" Oct 31 03:22:25.081673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:25.081621 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:25.602470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:25.601749 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:22:25.716725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:25.716690 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:25.888500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:25.888314 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:26.136980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:26.136907 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:26.975336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:26.975289 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:22:27.058507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:27.058470 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:28.044440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:28.044107 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:28.199440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:28.194699 4556 scope.go:122] "RemoveContainer" containerID="81476fa6fb69124bcc2d0644883f3ec4bb3fee72d888275f52be11b4a858ee74" Oct 31 03:22:29.101695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:29.101650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:29.801985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:29.801949 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:30.082106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:30.082021 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:30.241144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:30.241109 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:30.602969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:30.602914 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" containerID="cri-o://92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931" gracePeriod=25 Oct 31 03:22:30.658484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:30.658394 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:31.507995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:31.507960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:32.030883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.030840 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:32.202125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.202089 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:32.300610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.300510 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:22:32.300795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.300762 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" containerID="cri-o://026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d" gracePeriod=30 Oct 31 03:22:32.390077 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.390041 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:32.566720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.562634 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:32.905261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:32.905221 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:22:33.492332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:33.492304 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:34.358676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:34.358635 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:35.048943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.048902 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:35.150940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.150893 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" containerID="cri-o://1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f" gracePeriod=45 Oct 31 03:22:35.436348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.436311 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:22:35.504092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504044 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.504305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504128 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.504305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504264 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\" (UniqueName: \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.504453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504310 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\" (UniqueName: \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.504453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504357 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.504620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.504550 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\") pod \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\" (UID: \"0846676f-1acd-4a0e-9c38-3dcf0243ba74\") " Oct 31 03:22:35.505760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.505726 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.505760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.505716 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.505994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.505806 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.508052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.508024 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:35.508553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.508531 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:35.508929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.508908 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr" pod "0846676f-1acd-4a0e-9c38-3dcf0243ba74" (UID: "0846676f-1acd-4a0e-9c38-3dcf0243ba74"). InnerVolumeSpecName "kube-api-access-pwhqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:35.607376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607281 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhqr\" (UniqueName: \"kubernetes.io/projected/0846676f-1acd-4a0e-9c38-3dcf0243ba74-kube-api-access-pwhqr\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.607376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607326 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.607376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607342 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.607376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607355 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.607376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607369 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.607719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.607382 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0846676f-1acd-4a0e-9c38-3dcf0243ba74-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:35.876655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:35.876569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:36.248510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.248475 4556 generic.go:356] "Generic (PLEG): container finished" podID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerID="1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f" exitCode=0 Oct 31 03:22:36.248690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.248541 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" Oct 31 03:22:36.248690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.248573 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" event={"ID":"0846676f-1acd-4a0e-9c38-3dcf0243ba74","Type":"ContainerDied","Data":"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f"} Oct 31 03:22:36.248690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.248624 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" event={"ID":"0846676f-1acd-4a0e-9c38-3dcf0243ba74","Type":"ContainerDied","Data":"d83f77cad6b3ffeec3b9599191475ff286ac50e7de984d69afcad785d405ce04"} Oct 31 03:22:36.248690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.248647 4556 scope.go:122] "RemoveContainer" containerID="1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f" Oct 31 03:22:36.269914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.269883 4556 scope.go:122] "RemoveContainer" containerID="1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f" Oct 31 03:22:36.270357 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:36.270325 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f\": container with ID starting with 1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f not found: ID does not exist" containerID="1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f" Oct 31 03:22:36.270487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.270368 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f"} err="failed to get container status \"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f\": rpc error: code = NotFound desc = could not find container \"1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f\": container with ID starting with 1b701e0e35668f30655f18650bd3003ff5a5327af5e01bf6bc0c8281b3e00f9f not found: ID does not exist" Oct 31 03:22:36.282985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.282949 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:22:36.286519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.286488 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm"] Oct 31 03:22:36.425761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.425713 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d8daf/distinct-2-w6-7d8daf-kserve-6b6f84f7bd-hzsrm" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:22:36.552101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:36.551933 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.046171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.046132 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.220347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.220303 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.402503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.402467 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.588369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.588289 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:22:37.741322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741280 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741337 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741366 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741499 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741603 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741714 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:37.741789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741771 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741800 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741831 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:37.741900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741836 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.741900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.741895 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\") pod \"a05d12c4-a22d-4420-afde-9c49f1361d31\" (UID: \"a05d12c4-a22d-4420-afde-9c49f1361d31\") " Oct 31 03:22:37.742130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742059 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:37.742130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742078 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:37.742474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742382 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:37.742907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742857 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.742907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742890 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.742907 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742906 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d12c4-a22d-4420-afde-9c49f1361d31-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.743202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742922 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.743202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.742939 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.743960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.743928 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:22:37.744232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.744210 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:37.744319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.744267 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "kube-api-access-vxmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:37.744498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.744478 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token" pod "a05d12c4-a22d-4420-afde-9c49f1361d31" (UID: "a05d12c4-a22d-4420-afde-9c49f1361d31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:37.844856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.844746 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.844856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.844785 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.844856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.844798 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:37.844856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:37.844810 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmlm\" (UniqueName: \"kubernetes.io/projected/a05d12c4-a22d-4420-afde-9c49f1361d31-kube-api-access-vxmlm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:38.132168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.131760 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:38.261108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.261070 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" path="/var/lib/kubelet/pods/0846676f-1acd-4a0e-9c38-3dcf0243ba74/volumes" Oct 31 03:22:38.299329 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.299283 4556 generic.go:356] "Generic (PLEG): container finished" podID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerID="026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d" exitCode=0 Oct 31 03:22:38.299532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.299345 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" Oct 31 03:22:38.299532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.299366 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerDied","Data":"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d"} Oct 31 03:22:38.299532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.299486 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh" event={"ID":"a05d12c4-a22d-4420-afde-9c49f1361d31","Type":"ContainerDied","Data":"30f58b635655d23834ab12716dec4f7361f9b55ca8f063e2f2039765e5168b6a"} Oct 31 03:22:38.299532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.299522 4556 scope.go:122] "RemoveContainer" containerID="026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d" Oct 31 03:22:38.310822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.310799 4556 scope.go:122] "RemoveContainer" containerID="38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf" Oct 31 03:22:38.318401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.318367 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:22:38.323655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.323628 4556 scope.go:122] "RemoveContainer" containerID="026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d" Oct 31 03:22:38.323829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.323806 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bbbc81-openshift-default-75ccc754c9-cfnjh"] Oct 31 03:22:38.323961 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:38.323943 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d\": container with ID starting with 026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d not found: ID does not exist" containerID="026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d" Oct 31 03:22:38.324005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.323970 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d"} err="failed to get container status \"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d\": rpc error: code = NotFound desc = could not find container \"026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d\": container with ID starting with 026bc6972cca27c2125f3a0cc515ec4cf820fe91660fa997ef0d1bb29ee9bf3d not found: ID does not exist" Oct 31 03:22:38.324005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.323987 4556 scope.go:122] "RemoveContainer" containerID="38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf" Oct 31 03:22:38.324215 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:38.324196 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf\": container with ID starting with 38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf not found: ID does not exist" containerID="38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf" Oct 31 03:22:38.324253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.324222 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf"} err="failed to get container status \"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf\": rpc error: code = NotFound desc = could not find container \"38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf\": container with ID starting with 38f98c8e8590d63314833d320dca39373bf0e2c328212062c5794a6cc9fad2bf not found: ID does not exist" Oct 31 03:22:38.990644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:38.990596 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:39.644951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:39.644918 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:40.258961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:40.258919 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" path="/var/lib/kubelet/pods/a05d12c4-a22d-4420-afde-9c49f1361d31/volumes" Oct 31 03:22:40.331236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:40.331197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:41.038809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:41.038773 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:41.880825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:41.880776 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:42.564674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:42.564602 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:42.905739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:42.905696 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:22:43.104387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:43.104353 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:43.284932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:43.284816 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:43.492229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:43.492163 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:44.407463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:44.407406 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:45.268799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:45.268760 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:45.938972 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:45.938931 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:22:46.123355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:46.122230 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:46.862099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:46.862030 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:47.551066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:47.551029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:48.336792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:48.336725 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:49.044961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:49.044922 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:49.959631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:49.959592 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:50.521149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:50.521106 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:50.710447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:50.710366 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:50.903942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:50.903903 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:50.963339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:50.963309 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:22:51.076737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.076691 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\") pod \"2da57060-3c69-42e2-b7c7-05047706a00a\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " Oct 31 03:22:51.077025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.077004 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\" (UniqueName: \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\") pod \"2da57060-3c69-42e2-b7c7-05047706a00a\" (UID: \"2da57060-3c69-42e2-b7c7-05047706a00a\") " Oct 31 03:22:51.078893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.078861 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume" pod "2da57060-3c69-42e2-b7c7-05047706a00a" (UID: "2da57060-3c69-42e2-b7c7-05047706a00a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:51.080224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.080195 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:51.081265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.081234 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n" pod "2da57060-3c69-42e2-b7c7-05047706a00a" (UID: "2da57060-3c69-42e2-b7c7-05047706a00a"). InnerVolumeSpecName "kube-api-access-62q8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:51.179542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.179504 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62q8n\" (UniqueName: \"kubernetes.io/projected/2da57060-3c69-42e2-b7c7-05047706a00a-kube-api-access-62q8n\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.179542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.179534 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da57060-3c69-42e2-b7c7-05047706a00a-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:51.435321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.435290 4556 generic.go:356] "Generic (PLEG): container finished" podID="2da57060-3c69-42e2-b7c7-05047706a00a" containerID="92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931" exitCode=0 Oct 31 03:22:51.435550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.435332 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" Oct 31 03:22:51.435550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.435369 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" event={"ID":"2da57060-3c69-42e2-b7c7-05047706a00a","Type":"ContainerDied","Data":"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931"} Oct 31 03:22:51.435550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.435442 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q" event={"ID":"2da57060-3c69-42e2-b7c7-05047706a00a","Type":"ContainerDied","Data":"cee8dc394631d03ae836c822728a5721b79499cc67cf03c1d6e53608abeae6d7"} Oct 31 03:22:51.435550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.435466 4556 scope.go:122] "RemoveContainer" containerID="92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931" Oct 31 03:22:51.446996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.446972 4556 scope.go:122] "RemoveContainer" containerID="92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931" Oct 31 03:22:51.447369 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:22:51.447342 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931\": container with ID starting with 92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931 not found: ID does not exist" containerID="92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931" Oct 31 03:22:51.447493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.447379 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931"} err="failed to get container status \"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931\": rpc error: code = NotFound desc = could not find container \"92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931\": container with ID starting with 92051dce2b00892e3f44a617cb517ef033c037d031a0a10bf5eaf525900c0931 not found: ID does not exist" Oct 31 03:22:51.450623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.450595 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:51.460673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.460633 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:22:51.466875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.466839 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7d8daf-7bc48d5f9-5gt6q"] Oct 31 03:22:51.988574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:51.988525 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:52.260985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:52.260884 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" path="/var/lib/kubelet/pods/2da57060-3c69-42e2-b7c7-05047706a00a/volumes" Oct 31 03:22:52.736019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:52.735956 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:53.445168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:53.445121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:54.127981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:54.127917 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:54.821899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:54.821807 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:55.546884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:55.546829 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:56.064889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.064846 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:22:56.065776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.065738 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" podUID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" containerName="maas-api" containerID="cri-o://fec643931cddc172202930556e002693660fc94e7676bc49d6828bb1bf3d400f" gracePeriod=30 Oct 31 03:22:56.161163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.161121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:56.581852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.581799 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:56.590574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.590538 4556 generic.go:356] "Generic (PLEG): container finished" podID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" containerID="fec643931cddc172202930556e002693660fc94e7676bc49d6828bb1bf3d400f" exitCode=0 Oct 31 03:22:56.590771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.590729 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" event={"ID":"7d1e5758-cd32-48ae-98ea-d6bc07457dc3","Type":"ContainerDied","Data":"fec643931cddc172202930556e002693660fc94e7676bc49d6828bb1bf3d400f"} Oct 31 03:22:56.630873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.630841 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:22:56.670269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.670227 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\") pod \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " Oct 31 03:22:56.670580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.670426 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\") pod \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " Oct 31 03:22:56.670694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.670626 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\" (UniqueName: \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\") pod \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " Oct 31 03:22:56.670694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.670686 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\") pod \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\" (UID: \"7d1e5758-cd32-48ae-98ea-d6bc07457dc3\") " Oct 31 03:22:56.670819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.670688 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca" pod "7d1e5758-cd32-48ae-98ea-d6bc07457dc3" (UID: "7d1e5758-cd32-48ae-98ea-d6bc07457dc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:56.671321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.671298 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:56.673464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.673424 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6" pod "7d1e5758-cd32-48ae-98ea-d6bc07457dc3" (UID: "7d1e5758-cd32-48ae-98ea-d6bc07457dc3"). InnerVolumeSpecName "kube-api-access-wfrp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:56.677647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.677602 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs" pod "7d1e5758-cd32-48ae-98ea-d6bc07457dc3" (UID: "7d1e5758-cd32-48ae-98ea-d6bc07457dc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:56.678196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.678158 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls" pod "7d1e5758-cd32-48ae-98ea-d6bc07457dc3" (UID: "7d1e5758-cd32-48ae-98ea-d6bc07457dc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:56.772822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.772778 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:56.772822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.772823 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrp6\" (UniqueName: \"kubernetes.io/projected/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-kube-api-access-wfrp6\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:56.773068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.772842 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d1e5758-cd32-48ae-98ea-d6bc07457dc3-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:56.783281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.783243 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:56.980626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:56.980594 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:57.203477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.197720 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:57.393493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.387910 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:57.629401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.625598 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" event={"ID":"7d1e5758-cd32-48ae-98ea-d6bc07457dc3","Type":"ContainerDied","Data":"591327e37afdabe333c7c9a19c45deb5a08b6dc89dfb6522da00ac922fb943fd"} Oct 31 03:22:57.629401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.625689 4556 scope.go:122] "RemoveContainer" containerID="fec643931cddc172202930556e002693660fc94e7676bc49d6828bb1bf3d400f" Oct 31 03:22:57.629401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.625847 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb" Oct 31 03:22:57.654479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.654275 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:57.767093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.766990 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:22:57.775501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.772506 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7d8daf-7474bb54f5-8rkpb"] Oct 31 03:22:57.819827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:57.819785 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:58.037743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.037639 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4"] Oct 31 03:22:58.041044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.041013 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb496ddc-7tkc4"] Oct 31 03:22:58.214134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.214090 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:58.259818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.259783 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7" path="/var/lib/kubelet/pods/0d03f1f8-7dbc-4749-87f4-e3a3ed69f1b7/volumes" Oct 31 03:22:58.260370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.260346 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" path="/var/lib/kubelet/pods/7d1e5758-cd32-48ae-98ea-d6bc07457dc3/volumes" Oct 31 03:22:58.915130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:58.913334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:59.788638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:22:59.788336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:00.634960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:00.634916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:00.815688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:00.815639 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:01.009387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:01.006282 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:01.205994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:01.205958 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:01.375919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:01.375827 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:01.525629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:01.525594 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:02.559445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:02.556084 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:03.444454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:03.444363 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:04.172930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.172822 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:04.329019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.328981 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:04.487648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.487549 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:23:04.487863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.487829 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" containerID="cri-o://537ffddc5550ffddf3055cbc77492f3962a4d776e6d8103d4adc100d787feff8" gracePeriod=30 Oct 31 03:23:04.493104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.493063 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:23:04.493352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.493325 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" containerID="cri-o://201dcd9d20a180a539a64b4ae9ecb5adc9dc9016635c89adc6503630222b6f28" gracePeriod=30 Oct 31 03:23:04.559993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:04.559948 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:05.226009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:05.225971 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:05.785328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:05.785286 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:05.785562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:05.785359 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:05.949454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:05.949372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:06.044666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:06.044565 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:23:06.764267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:06.764236 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:06.945460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:06.945251 4556 patch_prober.go:29] interesting pod/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:23:06.945793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:06.945747 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:23:07.661009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:07.660972 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:08.319552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:08.319514 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:08.463988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:08.463951 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:09.021288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.021251 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:09.432687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.432644 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:09.771369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.768933 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:09.775580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.775204 4556 generic.go:356] "Generic (PLEG): container finished" podID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerID="537ffddc5550ffddf3055cbc77492f3962a4d776e6d8103d4adc100d787feff8" exitCode=0 Oct 31 03:23:09.775580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.775510 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" event={"ID":"fccb95ac-a041-4acf-a984-d2cda2d08cc3","Type":"ContainerDied","Data":"537ffddc5550ffddf3055cbc77492f3962a4d776e6d8103d4adc100d787feff8"} Oct 31 03:23:09.778168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.778041 4556 generic.go:356] "Generic (PLEG): container finished" podID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerID="201dcd9d20a180a539a64b4ae9ecb5adc9dc9016635c89adc6503630222b6f28" exitCode=0 Oct 31 03:23:09.778168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.778089 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" event={"ID":"d7c23684-5b7f-4163-9bdc-1b3f164fc262","Type":"ContainerDied","Data":"201dcd9d20a180a539a64b4ae9ecb5adc9dc9016635c89adc6503630222b6f28"} Oct 31 03:23:09.844327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.844285 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:23:09.880940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.880887 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881015 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881085 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881156 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881191 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881280 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881385 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881557 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881634 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\") pod \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\" (UID: \"d7c23684-5b7f-4163-9bdc-1b3f164fc262\") " Oct 31 03:23:09.881885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.881858 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.882228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.882029 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.882471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.882447 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.882553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.882476 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.883048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.883024 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.883246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.883220 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.883509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.883479 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:09.885241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.884692 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:09.885744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.885699 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.885872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.885847 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "kube-api-access-wr5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.887799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.887771 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy" pod "d7c23684-5b7f-4163-9bdc-1b3f164fc262" (UID: "d7c23684-5b7f-4163-9bdc-1b3f164fc262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.915347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.915310 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:23:09.983732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.983625 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.983732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.983718 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.983993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.983771 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.983993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.983817 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.983993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.983908 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.984169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984033 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.984169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984125 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.984283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984237 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.984336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984292 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\") pod \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\" (UID: \"fccb95ac-a041-4acf-a984-d2cda2d08cc3\") " Oct 31 03:23:09.984523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984492 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.984687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984652 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.984911 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984884 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.985005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.984979 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:09.985276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985241 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.985490 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985352 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985502 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985523 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985537 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985550 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985564 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c23684-5b7f-4163-9bdc-1b3f164fc262-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985577 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985589 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985602 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c23684-5b7f-4163-9bdc-1b3f164fc262-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.985616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985616 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.986213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.985632 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/d7c23684-5b7f-4163-9bdc-1b3f164fc262-kube-api-access-wr5wb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:09.986601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.986538 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:09.986963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.986942 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:09.987221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.987195 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "kube-api-access-t5rvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:09.988016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:09.987992 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token" pod "fccb95ac-a041-4acf-a984-d2cda2d08cc3" (UID: "fccb95ac-a041-4acf-a984-d2cda2d08cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:10.087014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.086972 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.087014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.087010 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rvj\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-kube-api-access-t5rvj\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.087014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.087021 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.087279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.087030 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fccb95ac-a041-4acf-a984-d2cda2d08cc3-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.087279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.087040 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fccb95ac-a041-4acf-a984-d2cda2d08cc3-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:10.202511 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.202469 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:10.388829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.388672 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:10.786187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.786108 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" Oct 31 03:23:10.786717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.786106 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7" event={"ID":"d7c23684-5b7f-4163-9bdc-1b3f164fc262","Type":"ContainerDied","Data":"5eb9af0537db25baa485cd94ca8217dec62bf758ba84cb79dd995bc3517fb4df"} Oct 31 03:23:10.786717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.786247 4556 scope.go:122] "RemoveContainer" containerID="201dcd9d20a180a539a64b4ae9ecb5adc9dc9016635c89adc6503630222b6f28" Oct 31 03:23:10.787870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.787846 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" event={"ID":"fccb95ac-a041-4acf-a984-d2cda2d08cc3","Type":"ContainerDied","Data":"544f4ba23f51aff157c2039419743075f1580975b12382d3f49ef6219483e93c"} Oct 31 03:23:10.787942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.787931 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5" Oct 31 03:23:10.798091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.798066 4556 scope.go:122] "RemoveContainer" containerID="537ffddc5550ffddf3055cbc77492f3962a4d776e6d8103d4adc100d787feff8" Oct 31 03:23:10.812196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.812164 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:23:10.815868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.815837 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-45st7"] Oct 31 03:23:10.826536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.826506 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:23:10.830133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:10.830103 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d8daf-openshift-default-56ddb7845-tqtz5"] Oct 31 03:23:11.125598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:11.125435 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:11.920471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:11.914566 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:12.159354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:12.159320 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:12.438770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:12.438718 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:12.450772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:12.450132 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" path="/var/lib/kubelet/pods/d7c23684-5b7f-4163-9bdc-1b3f164fc262/volumes" Oct 31 03:23:12.451191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:12.451160 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" path="/var/lib/kubelet/pods/fccb95ac-a041-4acf-a984-d2cda2d08cc3/volumes" Oct 31 03:23:12.751739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:12.751631 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:13.437105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:13.435690 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.198196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:14.198148 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.515455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:14.515341 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:23:14.515955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:14.515596 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" containerID="cri-o://68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700" gracePeriod=30 Oct 31 03:23:14.811024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:14.810927 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.971344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:14.971309 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:15.120307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:15.120215 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:15.269189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:15.269111 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:15.603933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:15.603901 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:15.939751 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:23:15.939705 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:23:15.998868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:15.998834 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:16.720901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:16.720867 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:17.574371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:17.574317 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:18.221796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:18.221720 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:18.936904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:18.936865 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:19.648053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.648016 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:19.827577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.827546 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:23:19.854968 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.854929 4556 generic.go:356] "Generic (PLEG): container finished" podID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerID="68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700" exitCode=0 Oct 31 03:23:19.855165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.854992 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" Oct 31 03:23:19.855165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.855003 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" event={"ID":"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53","Type":"ContainerDied","Data":"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700"} Oct 31 03:23:19.855165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.855074 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2" event={"ID":"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53","Type":"ContainerDied","Data":"ddaab44e9a6cce99b08f521efe3afe82fa132537fc13372f80afccf57e31ed02"} Oct 31 03:23:19.855165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.855099 4556 scope.go:122] "RemoveContainer" containerID="68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700" Oct 31 03:23:19.867311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.867285 4556 scope.go:122] "RemoveContainer" containerID="68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700" Oct 31 03:23:19.867769 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:23:19.867735 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700\": container with ID starting with 68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700 not found: ID does not exist" containerID="68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700" Oct 31 03:23:19.867897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.867781 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700"} err="failed to get container status \"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700\": rpc error: code = NotFound desc = could not find container \"68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700\": container with ID starting with 68ce3bab1bbc870914288cc91f883172be0b63f87fee60145f17b03246f93700 not found: ID does not exist" Oct 31 03:23:19.969600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969557 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.969893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969612 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.969893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969748 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.969893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969864 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969896 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969936 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.969987 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970059 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970097 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970102 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\") pod \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\" (UID: \"27b9ef32-7ac0-4758-a7a6-e9537ebd1e53\") " Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970334 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970357 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970403 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970913 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970937 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970951 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:19.970986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.970965 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:19.971739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.971031 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:19.972318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.972289 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "kube-api-access-gp8m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:19.972549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.972524 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:19.972660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.972639 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:19.973977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:19.973955 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token" pod "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" (UID: "27b9ef32-7ac0-4758-a7a6-e9537ebd1e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:20.072972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.072868 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp8m9\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-kube-api-access-gp8m9\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:20.072972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.072907 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:20.072972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.072922 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:20.072972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.072934 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:20.072972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.072946 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:20.179168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.179125 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:23:20.183161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.183125 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-ll4p2"] Oct 31 03:23:20.258537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.258502 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" path="/var/lib/kubelet/pods/27b9ef32-7ac0-4758-a7a6-e9537ebd1e53/volumes" Oct 31 03:23:20.465759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:20.465707 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:21.290156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:21.290102 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:22.041698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:22.041653 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:22.627884 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:22.627842 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:23:22.755626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:22.755582 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:23.449608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:23.449571 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:23.598010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:23.597976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:23.724827 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:23.724743 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:23:23.725243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:23.725035 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" containerID="cri-o://7935b64b2ca0ed07ad834fc46c36cd79228a1354100aa436f061e94ec16614f4" gracePeriod=30 Oct 31 03:23:23.940477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:23.938658 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:24.221174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:24.221111 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:24.376314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:24.376271 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:25.213735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:25.213695 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:26.001474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:26.001433 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:26.749311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:26.749267 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:27.645522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:27.645482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:28.337526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:28.337484 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:28.995155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:28.995072 4556 generic.go:356] "Generic (PLEG): container finished" podID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerID="7935b64b2ca0ed07ad834fc46c36cd79228a1354100aa436f061e94ec16614f4" exitCode=0 Oct 31 03:23:28.995155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:28.995142 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerDied","Data":"7935b64b2ca0ed07ad834fc46c36cd79228a1354100aa436f061e94ec16614f4"} Oct 31 03:23:28.995390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:28.995206 4556 scope.go:122] "RemoveContainer" containerID="769d6fafdab299030aebc27bb6dc5bb8e8f6562186848fec51ab34cfd13c0222" Oct 31 03:23:29.022347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.022321 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:23:29.038721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.037546 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:29.120739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120691 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.120979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120775 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.120979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120850 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.120979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120885 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.120979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120934 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.120979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.120962 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.121269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.121012 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.121269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.121050 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.121269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.121135 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:29.121269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.121146 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\") pod \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\" (UID: \"07829292-1d8e-4032-91e6-6fa8d3fffcb3\") " Oct 31 03:23:29.121925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.121895 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:29.122181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122154 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:29.122320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122302 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:29.122545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122510 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:29.122846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122824 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122847 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122860 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122875 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.122889 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.124158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.124136 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:29.124338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.124314 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:29.124496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.124476 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "kube-api-access-mbptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:29.124843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.124822 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token" pod "07829292-1d8e-4032-91e6-6fa8d3fffcb3" (UID: "07829292-1d8e-4032-91e6-6fa8d3fffcb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:29.225042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.225004 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbptb\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-kube-api-access-mbptb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.225042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.225039 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.225042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.225050 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.225328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.225059 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07829292-1d8e-4032-91e6-6fa8d3fffcb3-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:29.257869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.257763 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:23:29.485493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.485454 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:29.931585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.931533 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:23:29.931849 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.931815 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" containerID="cri-o://853c3cfa2699256e140e4f54c4005e379ee3a3fe7b7f6822abb491da4387a596" gracePeriod=30 Oct 31 03:23:29.938720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:29.938686 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:30.012145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.012098 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" event={"ID":"07829292-1d8e-4032-91e6-6fa8d3fffcb3","Type":"ContainerDied","Data":"ec3b48cf6cf48044b453a4ca535e2acfb53e06ceff8d135280c8490353277896"} Oct 31 03:23:30.012358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.012229 4556 scope.go:122] "RemoveContainer" containerID="7935b64b2ca0ed07ad834fc46c36cd79228a1354100aa436f061e94ec16614f4" Oct 31 03:23:30.012551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.012528 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q" Oct 31 03:23:30.053525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.053468 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:23:30.056766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.056724 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3fc2d0-openshift-default-d746bd677-f686q"] Oct 31 03:23:30.260990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.260902 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" path="/var/lib/kubelet/pods/07829292-1d8e-4032-91e6-6fa8d3fffcb3/volumes" Oct 31 03:23:30.631041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:30.630907 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:31.423765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:31.423711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:32.242485 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:32.242376 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:33.018463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:33.018405 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:33.727425 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:33.725538 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:34.509132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:34.509086 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:34.703434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:34.703386 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:34.841715 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:34.841604 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:35.056137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.056100 4556 generic.go:356] "Generic (PLEG): container finished" podID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerID="853c3cfa2699256e140e4f54c4005e379ee3a3fe7b7f6822abb491da4387a596" exitCode=0 Oct 31 03:23:35.056307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.056176 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" event={"ID":"278bcb1f-859e-46eb-9cf7-8839fb4376cf","Type":"ContainerDied","Data":"853c3cfa2699256e140e4f54c4005e379ee3a3fe7b7f6822abb491da4387a596"} Oct 31 03:23:35.178486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.178451 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:23:35.212542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212510 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212555 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212625 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212740 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212783 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212818 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.212982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.212911 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.213092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.213012 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.213092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.213043 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\") pod \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\" (UID: \"278bcb1f-859e-46eb-9cf7-8839fb4376cf\") " Oct 31 03:23:35.213638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.213609 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:35.213822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.213791 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:35.214051 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.214031 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:35.214182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.214155 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:35.214382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.214354 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:35.247260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.243491 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:23:35.247260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.243527 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:35.247260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.243549 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "kube-api-access-tzdpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:35.247260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.243560 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy" pod "278bcb1f-859e-46eb-9cf7-8839fb4376cf" (UID: "278bcb1f-859e-46eb-9cf7-8839fb4376cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:35.315728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.315652 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319367 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319404 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319437 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdpt\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-kube-api-access-tzdpt\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319452 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319466 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319487 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319508 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319527 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:35.319612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:35.319544 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278bcb1f-859e-46eb-9cf7-8839fb4376cf-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:36.064973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.064942 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" Oct 31 03:23:36.065403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.064973 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88" event={"ID":"278bcb1f-859e-46eb-9cf7-8839fb4376cf","Type":"ContainerDied","Data":"17f3748c443e75e3ffad5ac64e3126b11d7b55ea6b7b87a1d4edac057d0e41a1"} Oct 31 03:23:36.065403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.065037 4556 scope.go:122] "RemoveContainer" containerID="853c3cfa2699256e140e4f54c4005e379ee3a3fe7b7f6822abb491da4387a596" Oct 31 03:23:36.089127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.089087 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:23:36.093613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.093581 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-vss88"] Oct 31 03:23:36.151912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.151878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:36.259807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.259773 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" path="/var/lib/kubelet/pods/278bcb1f-859e-46eb-9cf7-8839fb4376cf/volumes" Oct 31 03:23:36.854825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:36.854753 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:37.629947 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:37.629899 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:38.301743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:38.301676 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:39.039819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:39.039672 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:39.781970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:39.781931 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:40.459489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:40.459407 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:41.313637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:41.313600 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:42.010047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:42.010001 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:42.775212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:42.775174 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:43.042519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:43.042163 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:43.770904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:43.770817 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:44.280141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.280067 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" containerID="cri-o://c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b" gracePeriod=45 Oct 31 03:23:44.522518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.520270 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:44.584656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.584625 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:23:44.678093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678057 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678154 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678263 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678278 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678307 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678381 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\" (UniqueName: \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\") pod \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\" (UID: \"60edd9f0-94a4-4219-9bc9-f73a1044af7f\") " Oct 31 03:23:44.678499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678437 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.678646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678539 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.678880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.678855 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.679035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.679018 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.679114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.679035 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.679114 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.679045 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.680890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.680862 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "kube-api-access-v42d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:44.681078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.681059 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:23:44.681078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.681067 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs" pod "60edd9f0-94a4-4219-9bc9-f73a1044af7f" (UID: "60edd9f0-94a4-4219-9bc9-f73a1044af7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:44.780644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.780605 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60edd9f0-94a4-4219-9bc9-f73a1044af7f-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.780644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.780641 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60edd9f0-94a4-4219-9bc9-f73a1044af7f-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:44.780859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:44.780655 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42d2\" (UniqueName: \"kubernetes.io/projected/60edd9f0-94a4-4219-9bc9-f73a1044af7f-kube-api-access-v42d2\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:45.129701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.129660 4556 generic.go:356] "Generic (PLEG): container finished" podID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerID="c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b" exitCode=0 Oct 31 03:23:45.129701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.129704 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" Oct 31 03:23:45.129996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.129749 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" event={"ID":"60edd9f0-94a4-4219-9bc9-f73a1044af7f","Type":"ContainerDied","Data":"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b"} Oct 31 03:23:45.129996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.129803 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf" event={"ID":"60edd9f0-94a4-4219-9bc9-f73a1044af7f","Type":"ContainerDied","Data":"22a9a5249c63e5715a461bfb988616052d19c0a4ddca84d396e03d4e1de1121a"} Oct 31 03:23:45.129996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.129824 4556 scope.go:122] "RemoveContainer" containerID="c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b" Oct 31 03:23:45.142698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.142667 4556 scope.go:122] "RemoveContainer" containerID="c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b" Oct 31 03:23:45.143216 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:23:45.143186 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b\": container with ID starting with c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b not found: ID does not exist" containerID="c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b" Oct 31 03:23:45.143335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.143226 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b"} err="failed to get container status \"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b\": rpc error: code = NotFound desc = could not find container \"c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b\": container with ID starting with c735baa71fe4f82af79c4c68c19736b2ef8aaa2f3b965565d2332f12d5c3a55b not found: ID does not exist" Oct 31 03:23:45.157844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.157792 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:23:45.162848 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.162811 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-85cb59/simulator-w2-85cb59-kserve-74f449fdb9-sncmf"] Oct 31 03:23:45.253553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:45.253514 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:45.938464 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:23:45.938394 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:23:46.036471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:46.036391 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:46.258821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:46.258736 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" path="/var/lib/kubelet/pods/60edd9f0-94a4-4219-9bc9-f73a1044af7f/volumes" Oct 31 03:23:46.821738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:46.821545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:47.563775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:47.563711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:48.357535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:48.357486 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:49.115219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.115087 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:49.839438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.835267 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:49.851851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.851812 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:23:49.852932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852900 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" Oct 31 03:23:49.852932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852930 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852948 4556 cpu_manager.go:414] "containerMap: removing container" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852955 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852967 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852974 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852987 4556 cpu_manager.go:414] "containerMap: removing container" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.852994 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853011 4556 cpu_manager.go:414] "containerMap: removing container" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853018 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853033 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853040 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853051 4556 cpu_manager.go:414] "containerMap: removing container" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853058 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853075 4556 cpu_manager.go:414] "containerMap: removing container" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853081 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853091 4556 cpu_manager.go:414] "containerMap: removing container" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853098 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853109 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e014c245-4973-4be6-9968-2bb75c5fda94" containerName="maas-api" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853115 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e014c245-4973-4be6-9968-2bb75c5fda94" containerName="maas-api" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853127 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" containerName="payload-pre-processing" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853136 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" containerName="payload-pre-processing" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853152 4556 cpu_manager.go:414] "containerMap: removing container" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853160 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853170 4556 cpu_manager.go:414] "containerMap: removing container" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853177 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" Oct 31 03:23:49.853175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853187 4556 cpu_manager.go:414] "containerMap: removing container" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853194 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853214 4556 cpu_manager.go:414] "containerMap: removing container" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853222 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853233 4556 cpu_manager.go:414] "containerMap: removing container" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853240 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853251 4556 cpu_manager.go:414] "containerMap: removing container" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853258 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853272 4556 cpu_manager.go:414] "containerMap: removing container" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853278 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853294 4556 cpu_manager.go:414] "containerMap: removing container" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853302 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853313 4556 cpu_manager.go:414] "containerMap: removing container" podUID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" containerName="maas-api" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853319 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" containerName="maas-api" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853329 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853336 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853360 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.854455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.853368 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854540 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854572 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c23684-5b7f-4163-9bdc-1b3f164fc262" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854585 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854593 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da57060-3c69-42e2-b7c7-05047706a00a" containerName="payload-pre-processing" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854607 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854617 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854627 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854636 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f434bd-237c-4c9f-aa3d-af3599c7feeb" containerName="main" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854660 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccb95ac-a041-4acf-a984-d2cda2d08cc3" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854676 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="914c3f84-6dbe-4652-a97d-6125aa9db86d" containerName="main" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854685 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e014c245-4973-4be6-9968-2bb75c5fda94" containerName="maas-api" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854694 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e6036e-25e1-4103-ba8b-4d3f4048db2a" containerName="payload-pre-processing" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854704 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="198e0fdd-69de-4242-83f7-bee12900dc7c" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854712 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="278bcb1f-859e-46eb-9cf7-8839fb4376cf" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854720 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b9ef32-7ac0-4758-a7a6-e9537ebd1e53" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854732 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f9780b-1631-4c9f-bedd-df67d21f1d83" containerName="payload-pre-processing" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854741 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="263b74e5-a37c-4823-8232-317ea36efd3e" containerName="istio-proxy" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854749 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="60edd9f0-94a4-4219-9bc9-f73a1044af7f" containerName="main" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854757 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="59be27b7-2cc1-4119-8c9c-4964760df2a7" containerName="main" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854767 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="314bff69-cff9-4977-a24c-d9b7d65f0a40" containerName="payload-processing" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854777 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1e5758-cd32-48ae-98ea-d6bc07457dc3" containerName="maas-api" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854787 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0846676f-1acd-4a0e-9c38-3dcf0243ba74" containerName="main" Oct 31 03:23:49.855331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.854800 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05d12c4-a22d-4420-afde-9c49f1361d31" containerName="istio-proxy" Oct 31 03:23:49.860659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.860593 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.867806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.867772 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b79c13-openshift-default-dockercfg-wfw7v\"" Oct 31 03:23:49.868059 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.867782 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:23:49.959952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.959915 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.959952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.959961 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960087 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960152 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960247 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960319 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4wn\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960368 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960450 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:49.960512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:49.960498 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062299 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062398 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062522 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4wn\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062588 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062639 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062689 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062734 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062769 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.062814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062792 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.063093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062827 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.063093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.062855 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.063206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.063105 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.063206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.063134 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.063540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.063520 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.065144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.065119 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.065532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.065512 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.074072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.074042 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.074217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.074144 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4wn\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\") pod \"e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.192736 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.192701 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:50.481344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.481312 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:23:50.483339 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:23:50.483308 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee7dd0d_24dc_457c_ad21_38a84b07f1b9.slice/crio-b45574307543002443143c28ccfcb53f44201a295d5d872e3bef03014c26b2ce WatchSource:0}: Error finding container b45574307543002443143c28ccfcb53f44201a295d5d872e3bef03014c26b2ce: Status 404 returned error can't find the container with id b45574307543002443143c28ccfcb53f44201a295d5d872e3bef03014c26b2ce Oct 31 03:23:50.486260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.486220 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:23:50.486342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.486315 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:23:50.486388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.486359 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:23:50.537711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:50.535022 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:51.200963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:51.200925 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" event={"ID":"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9","Type":"ContainerStarted","Data":"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312"} Oct 31 03:23:51.201469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:51.200987 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" event={"ID":"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9","Type":"ContainerStarted","Data":"b45574307543002443143c28ccfcb53f44201a295d5d872e3bef03014c26b2ce"} Oct 31 03:23:51.227757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:51.227683 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podStartSLOduration=2.22765961 podStartE2EDuration="2.22765961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:51.222226232 +0000 UTC m=+1709.077243374" watchObservedRunningTime="2026-10-31 03:23:51.22765961 +0000 UTC m=+1709.082676758" Oct 31 03:23:51.312436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:51.312266 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.018678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.018643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.193301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.193086 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:52.195008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.194968 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 03:23:52.195184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.195038 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 03:23:52.594375 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.594332 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.746467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.745858 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:52.890500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:52.885974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:53.101323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:53.101285 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:53.205065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:53.205024 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 03:23:53.205384 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:53.205352 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 03:23:53.488495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:53.484941 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:53.822625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:53.822531 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:54.063409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:54.063378 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:54.194319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:54.194284 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 03:23:54.194660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:54.194615 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 03:23:54.498067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:54.497969 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:23:55.127519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:55.127479 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:55.197438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:55.193828 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 03:23:55.197438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:55.193908 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 03:23:56.005982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:56.005157 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:56.202143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:56.197325 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 31 03:23:56.202143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:56.197389 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 31 03:23:56.993581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:56.993527 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:57.381501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:57.381372 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:57.382084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:57.381894 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:57.382783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:57.382762 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:23:57.910174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:57.910131 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:58.702061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:58.702024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:59.137577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:59.137479 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:59.803224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:23:59.803187 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:00.524596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:00.524557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.314519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.314479 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.543873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.543830 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.687461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.687401 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:01.860991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.860943 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:24:01.861299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.861266 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" containerID="cri-o://58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52" gracePeriod=30 Oct 31 03:24:01.921406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:01.921360 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:02.258929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:02.258890 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:03.034943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:03.034862 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:03.833204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:03.833151 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:04.120638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:04.120539 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:24:04.264916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:04.264881 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:04.397406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:04.397253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:04.741233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:04.741191 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:05.521550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:05.521513 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:06.328510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.328466 4556 patch_prober.go:29] interesting pod/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:06.328707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.328555 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:06.432944 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.432909 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:06.823718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.823675 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:06.884861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.884819 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:24:06.885118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:06.885085 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" podUID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" containerName="maas-api" containerID="cri-o://5c7f13eb95ccc01826b02028670751a456d97926d8a49eb829fa4eb3c06cf867" gracePeriod=30 Oct 31 03:24:07.031012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.029113 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.162461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.162399 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:24:07.163819 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.163787 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316307 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316407 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316597 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316637 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316757 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316787 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316892 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316954 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.316994 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\") pod \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\" (UID: \"405f6ada-3a66-45f1-acd8-64f2729f4ba2\") " Oct 31 03:24:07.323519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.322213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.324255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.323843 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:07.324255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.324129 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.325532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.324638 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.325532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.324976 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.325532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.325319 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.369898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.354977 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:07.369898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.356585 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.369898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.356720 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:07.379277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.376700 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn" pod "405f6ada-3a66-45f1-acd8-64f2729f4ba2" (UID: "405f6ada-3a66-45f1-acd8-64f2729f4ba2"). InnerVolumeSpecName "kube-api-access-4ppkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.382783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.382626 4556 generic.go:356] "Generic (PLEG): container finished" podID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" containerID="5c7f13eb95ccc01826b02028670751a456d97926d8a49eb829fa4eb3c06cf867" exitCode=0 Oct 31 03:24:07.382783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.382701 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" event={"ID":"07f103c4-8022-4f07-a6be-b76dfa9f1ccc","Type":"ContainerDied","Data":"5c7f13eb95ccc01826b02028670751a456d97926d8a49eb829fa4eb3c06cf867"} Oct 31 03:24:07.385462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.384701 4556 generic.go:356] "Generic (PLEG): container finished" podID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerID="58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52" exitCode=0 Oct 31 03:24:07.385462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.384886 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" event={"ID":"405f6ada-3a66-45f1-acd8-64f2729f4ba2","Type":"ContainerDied","Data":"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52"} Oct 31 03:24:07.385462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.384942 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" event={"ID":"405f6ada-3a66-45f1-acd8-64f2729f4ba2","Type":"ContainerDied","Data":"f6d43b27ab5489842226d167bde2b38c414d8ffad96613b8e9466ac371dd60d3"} Oct 31 03:24:07.385462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.384974 4556 scope.go:122] "RemoveContainer" containerID="58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52" Oct 31 03:24:07.385462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.385140 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.417832 4556 scope.go:122] "RemoveContainer" containerID="58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419182 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419205 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419217 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419228 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419240 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ppkn\" (UniqueName: \"kubernetes.io/projected/405f6ada-3a66-45f1-acd8-64f2729f4ba2-kube-api-access-4ppkn\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419251 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419259 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419268 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405f6ada-3a66-45f1-acd8-64f2729f4ba2-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.420667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.419277 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405f6ada-3a66-45f1-acd8-64f2729f4ba2-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.421224 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:07.420687 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52\": container with ID starting with 58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52 not found: ID does not exist" containerID="58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52" Oct 31 03:24:07.421224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.420722 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52"} err="failed to get container status \"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52\": rpc error: code = NotFound desc = could not find container \"58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52\": container with ID starting with 58535c9b2cc88398e157e4697c3a56fb5933333f2dfff1bb30800ece28754d52 not found: ID does not exist" Oct 31 03:24:07.456053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.455127 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:07.481473 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.480542 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:24:07.487127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.487094 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-85cb59-openshift-default-6cf4786477-hwlff"] Oct 31 03:24:07.561692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.561665 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:24:07.620763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.620717 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\") pod \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " Oct 31 03:24:07.620976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.620784 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\") pod \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " Oct 31 03:24:07.620976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.620955 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\") pod \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " Oct 31 03:24:07.621111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.620996 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\" (UniqueName: \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\") pod \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\" (UID: \"07f103c4-8022-4f07-a6be-b76dfa9f1ccc\") " Oct 31 03:24:07.621244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.621205 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca" pod "07f103c4-8022-4f07-a6be-b76dfa9f1ccc" (UID: "07f103c4-8022-4f07-a6be-b76dfa9f1ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:07.621527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.621500 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.623198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.623174 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls" pod "07f103c4-8022-4f07-a6be-b76dfa9f1ccc" (UID: "07f103c4-8022-4f07-a6be-b76dfa9f1ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:07.623708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.623665 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq" pod "07f103c4-8022-4f07-a6be-b76dfa9f1ccc" (UID: "07f103c4-8022-4f07-a6be-b76dfa9f1ccc"). InnerVolumeSpecName "kube-api-access-mhzwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:07.624464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.624405 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs" pod "07f103c4-8022-4f07-a6be-b76dfa9f1ccc" (UID: "07f103c4-8022-4f07-a6be-b76dfa9f1ccc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:07.722456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.722366 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.722456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.722398 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzwq\" (UniqueName: \"kubernetes.io/projected/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-kube-api-access-mhzwq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.722456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.722409 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f103c4-8022-4f07-a6be-b76dfa9f1ccc-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:07.794094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:07.794048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:08.246438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.242597 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:08.266852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.266574 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" path="/var/lib/kubelet/pods/405f6ada-3a66-45f1-acd8-64f2729f4ba2/volumes" Oct 31 03:24:08.394019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.393988 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" Oct 31 03:24:08.394196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.393979 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz" event={"ID":"07f103c4-8022-4f07-a6be-b76dfa9f1ccc","Type":"ContainerDied","Data":"7d894642520ff64ea71eeba9c6e9f7d97fad996543cf4683a9920b3604375fdd"} Oct 31 03:24:08.394196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.394161 4556 scope.go:122] "RemoveContainer" containerID="5c7f13eb95ccc01826b02028670751a456d97926d8a49eb829fa4eb3c06cf867" Oct 31 03:24:08.413744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.413701 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:24:08.416712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:08.416681 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2bec7-567d8769b8-krfvz"] Oct 31 03:24:09.060479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.060011 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:09.121594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.121534 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" podUID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" containerName="payload-processing" containerID="cri-o://dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a" gracePeriod=25 Oct 31 03:24:09.394786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.394751 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:24:09.405120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.405085 4556 generic.go:356] "Generic (PLEG): container finished" podID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" containerID="dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a" exitCode=0 Oct 31 03:24:09.405302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.405131 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" Oct 31 03:24:09.405302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.405167 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" event={"ID":"cefa6a8e-9cfa-4809-83be-a2ba264b914c","Type":"ContainerDied","Data":"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a"} Oct 31 03:24:09.405302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.405218 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm" event={"ID":"cefa6a8e-9cfa-4809-83be-a2ba264b914c","Type":"ContainerDied","Data":"8e9a7e70a357800c48891ec76737db2c1e38b95aa4050fa65e1f227d4271a250"} Oct 31 03:24:09.405302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.405243 4556 scope.go:122] "RemoveContainer" containerID="dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a" Oct 31 03:24:09.418670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.418648 4556 scope.go:122] "RemoveContainer" containerID="dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a" Oct 31 03:24:09.419050 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:09.419014 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a\": container with ID starting with dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a not found: ID does not exist" containerID="dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a" Oct 31 03:24:09.419193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.419061 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a"} err="failed to get container status \"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a\": rpc error: code = NotFound desc = could not find container \"dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a\": container with ID starting with dacf636edb88fdb992d7776407da911546684fa831b0b4cd02a7f2b78e7ae80a not found: ID does not exist" Oct 31 03:24:09.446984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.446945 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\") pod \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " Oct 31 03:24:09.447459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.447398 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume" pod "cefa6a8e-9cfa-4809-83be-a2ba264b914c" (UID: "cefa6a8e-9cfa-4809-83be-a2ba264b914c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:09.447597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.447524 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefa6a8e-9cfa-4809-83be-a2ba264b914c-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.462570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.462517 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:09.548158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.548121 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\" (UniqueName: \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\") pod \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\" (UID: \"cefa6a8e-9cfa-4809-83be-a2ba264b914c\") " Oct 31 03:24:09.550446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.550394 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6" pod "cefa6a8e-9cfa-4809-83be-a2ba264b914c" (UID: "cefa6a8e-9cfa-4809-83be-a2ba264b914c"). InnerVolumeSpecName "kube-api-access-2fjj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:09.599545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.599506 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:09.650109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.650075 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjj6\" (UniqueName: \"kubernetes.io/projected/cefa6a8e-9cfa-4809-83be-a2ba264b914c-kube-api-access-2fjj6\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:09.727218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.727181 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:24:09.733765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.733730 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2bec7-56749545-dmrpm"] Oct 31 03:24:09.826869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:09.826386 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:10.241235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:10.241205 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:10.260288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:10.260244 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" path="/var/lib/kubelet/pods/07f103c4-8022-4f07-a6be-b76dfa9f1ccc/volumes" Oct 31 03:24:10.260784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:10.260760 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" path="/var/lib/kubelet/pods/cefa6a8e-9cfa-4809-83be-a2ba264b914c/volumes" Oct 31 03:24:10.519143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:10.519045 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:11.339004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:11.338974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:11.882777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:11.882732 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:12.232693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:12.232650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:13.396814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:13.396775 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:14.240776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:14.240724 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.015494 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:15.015451 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.480390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:15.480354 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.609931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:15.609864 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.861517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:15.861391 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.991017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:15.990976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:16.011863 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:16.011816 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:24:16.216538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:16.216503 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:16.350919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:16.350885 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:16.678868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:16.678831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:16.917560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:16.917518 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:17.730868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:17.730831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:18.455546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:18.455498 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:19.138664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:19.136020 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:19.911086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:19.911038 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:20.532496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:20.532458 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:20.672274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:20.672231 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:21.416647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:21.416609 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.129872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.129826 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.268598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.268563 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:24:22.278854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.278817 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:24:22.375795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.375733 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.575920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.575878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.905453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:22.905389 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:23.217057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:23.216968 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:24:23.611220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:23.611121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:24.342880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:24.341831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:25.300640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:25.300605 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:25.725053 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:25.724996 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.262262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.262230 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:24:26.262527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.262489 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerName="maas-api" containerID="cri-o://f0c77ae6c4a6dcf1745a5914a663a687675ad338d91a373ea2338cddab07861c" gracePeriod=30 Oct 31 03:24:26.315808 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.315765 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.356340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.356293 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.189:8443/health\": dial tcp 10.134.0.189:8443: connect: connection refused" Oct 31 03:24:26.431090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.431035 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:24:26.581633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.581462 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.606561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.606523 4556 generic.go:356] "Generic (PLEG): container finished" podID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerID="f0c77ae6c4a6dcf1745a5914a663a687675ad338d91a373ea2338cddab07861c" exitCode=0 Oct 31 03:24:26.607007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.606977 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" event={"ID":"ffa98cdf-5d23-4e37-8c63-6e8619231161","Type":"ContainerDied","Data":"f0c77ae6c4a6dcf1745a5914a663a687675ad338d91a373ea2338cddab07861c"} Oct 31 03:24:26.717553 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.717308 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.847752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.847654 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:24:26.852109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.852077 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.861465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.860604 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\" (UniqueName: \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\") pod \"ffa98cdf-5d23-4e37-8c63-6e8619231161\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " Oct 31 03:24:26.861465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.860739 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\") pod \"ffa98cdf-5d23-4e37-8c63-6e8619231161\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " Oct 31 03:24:26.861465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.860801 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\") pod \"ffa98cdf-5d23-4e37-8c63-6e8619231161\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " Oct 31 03:24:26.861465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.860866 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\") pod \"ffa98cdf-5d23-4e37-8c63-6e8619231161\" (UID: \"ffa98cdf-5d23-4e37-8c63-6e8619231161\") " Oct 31 03:24:26.881567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.881525 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca" pod "ffa98cdf-5d23-4e37-8c63-6e8619231161" (UID: "ffa98cdf-5d23-4e37-8c63-6e8619231161"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:26.887338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.887296 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls" pod "ffa98cdf-5d23-4e37-8c63-6e8619231161" (UID: "ffa98cdf-5d23-4e37-8c63-6e8619231161"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:26.890799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.890743 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq" pod "ffa98cdf-5d23-4e37-8c63-6e8619231161" (UID: "ffa98cdf-5d23-4e37-8c63-6e8619231161"). InnerVolumeSpecName "kube-api-access-xlrlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:26.903166 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.903110 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs" pod "ffa98cdf-5d23-4e37-8c63-6e8619231161" (UID: "ffa98cdf-5d23-4e37-8c63-6e8619231161"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:26.962870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.962829 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:26.962870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.962864 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa98cdf-5d23-4e37-8c63-6e8619231161-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:26.962870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.962874 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa98cdf-5d23-4e37-8c63-6e8619231161-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:26.963264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:26.962886 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrlq\" (UniqueName: \"kubernetes.io/projected/ffa98cdf-5d23-4e37-8c63-6e8619231161-kube-api-access-xlrlq\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.201138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.201096 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.276979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.276882 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" containerID="cri-o://0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16" gracePeriod=25 Oct 31 03:24:27.279725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.279628 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" containerID="cri-o://a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8" gracePeriod=25 Oct 31 03:24:27.347602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.347145 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:24:27.347602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.347573 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" podUID="9840d12a-89c6-44d4-976d-5b32edd8a23d" containerName="maas-api" containerID="cri-o://e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11" gracePeriod=30 Oct 31 03:24:27.432205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.432168 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.630670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.630536 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.659589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.647945 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" event={"ID":"ffa98cdf-5d23-4e37-8c63-6e8619231161","Type":"ContainerDied","Data":"823716230d371a57101eb1bcd9c82ea65f5d206b4d6a0c88bffbe73981cd391d"} Oct 31 03:24:27.659589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.648031 4556 scope.go:122] "RemoveContainer" containerID="f0c77ae6c4a6dcf1745a5914a663a687675ad338d91a373ea2338cddab07861c" Oct 31 03:24:27.659589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.648232 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m" Oct 31 03:24:27.726174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.726137 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:24:27.729238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.729168 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62bf00-5bcf88c57b-27t7m"] Oct 31 03:24:27.776045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.775856 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.872211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.871785 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:24:27.905732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.905694 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:27.961116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.961080 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:27.981309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.981275 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\") pod \"9840d12a-89c6-44d4-976d-5b32edd8a23d\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " Oct 31 03:24:27.981519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.981333 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\") pod \"9840d12a-89c6-44d4-976d-5b32edd8a23d\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " Oct 31 03:24:27.981519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.981391 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\" (UniqueName: \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\") pod \"9840d12a-89c6-44d4-976d-5b32edd8a23d\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " Oct 31 03:24:27.981611 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.981588 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") pod \"9840d12a-89c6-44d4-976d-5b32edd8a23d\" (UID: \"9840d12a-89c6-44d4-976d-5b32edd8a23d\") " Oct 31 03:24:27.982509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.981735 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca" pod "9840d12a-89c6-44d4-976d-5b32edd8a23d" (UID: "9840d12a-89c6-44d4-976d-5b32edd8a23d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:27.982509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.982148 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9840d12a-89c6-44d4-976d-5b32edd8a23d-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:27.985008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.984967 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls" pod "9840d12a-89c6-44d4-976d-5b32edd8a23d" (UID: "9840d12a-89c6-44d4-976d-5b32edd8a23d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:27.985234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.985214 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs" pod "9840d12a-89c6-44d4-976d-5b32edd8a23d" (UID: "9840d12a-89c6-44d4-976d-5b32edd8a23d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:27.985348 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:27.985314 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q" pod "9840d12a-89c6-44d4-976d-5b32edd8a23d" (UID: "9840d12a-89c6-44d4-976d-5b32edd8a23d"). InnerVolumeSpecName "kube-api-access-zm96q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:28.083370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.083318 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:28.083370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.083353 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9840d12a-89c6-44d4-976d-5b32edd8a23d-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:28.083370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.083364 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm96q\" (UniqueName: \"kubernetes.io/projected/9840d12a-89c6-44d4-976d-5b32edd8a23d-kube-api-access-zm96q\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:28.258582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.258496 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" path="/var/lib/kubelet/pods/ffa98cdf-5d23-4e37-8c63-6e8619231161/volumes" Oct 31 03:24:28.382052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.382016 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.455972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.455930 4556 scope.go:122] "RemoveContainer" containerID="9014270af38b59a3b5a5a8bd550cca99dcee2cdc5cc46d9b2d18b341887f4859" Oct 31 03:24:28.466650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.466628 4556 scope.go:122] "RemoveContainer" containerID="e09ab7de1a5c8232170baf1c84ad1bfb9cd30fa568d4bceda4a829edee193852" Oct 31 03:24:28.618090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.618029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.658164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.658128 4556 generic.go:356] "Generic (PLEG): container finished" podID="9840d12a-89c6-44d4-976d-5b32edd8a23d" containerID="e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11" exitCode=0 Oct 31 03:24:28.658360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.658199 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" Oct 31 03:24:28.658360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.658254 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" event={"ID":"9840d12a-89c6-44d4-976d-5b32edd8a23d","Type":"ContainerDied","Data":"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11"} Oct 31 03:24:28.658360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.658322 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq" event={"ID":"9840d12a-89c6-44d4-976d-5b32edd8a23d","Type":"ContainerDied","Data":"b22688eaeda44b98a8150094a0c69bb08ac03a22b9792e9fe6ad624f85a09da1"} Oct 31 03:24:28.658544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.658362 4556 scope.go:122] "RemoveContainer" containerID="e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11" Oct 31 03:24:28.681832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.681792 4556 scope.go:122] "RemoveContainer" containerID="e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11" Oct 31 03:24:28.681989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.681904 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:24:28.682255 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:28.682231 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11\": container with ID starting with e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11 not found: ID does not exist" containerID="e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11" Oct 31 03:24:28.682347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.682268 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11"} err="failed to get container status \"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11\": rpc error: code = NotFound desc = could not find container \"e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11\": container with ID starting with e24f7fcb39f3fa3550081b5cc0df9247c4cc55f2586d9d156296efb98791db11 not found: ID does not exist" Oct 31 03:24:28.684783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:28.684755 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7269ae-6f88c7484f-vjmzq"] Oct 31 03:24:29.313219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:29.312959 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.124393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:30.124354 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.258572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:30.258539 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9840d12a-89c6-44d4-976d-5b32edd8a23d" path="/var/lib/kubelet/pods/9840d12a-89c6-44d4-976d-5b32edd8a23d/volumes" Oct 31 03:24:30.917220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:30.917169 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:31.432395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:31.432351 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" containerID="cri-o://6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72" gracePeriod=25 Oct 31 03:24:31.637151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:31.637096 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:32.314486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:32.314408 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:33.031997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:33.030915 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:33.722106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:33.722068 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:34.528657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:34.521782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:35.223302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.223261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:35.363120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.363087 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:35.495519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.494734 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:35.718476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.718434 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:24:35.718707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.718663 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" containerID="cri-o://ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff" gracePeriod=30 Oct 31 03:24:35.729503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.729464 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:24:35.729735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.729687 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" containerID="cri-o://c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976" gracePeriod=30 Oct 31 03:24:35.738262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.738229 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:24:35.738522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.738456 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" containerID="cri-o://edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6" gracePeriod=30 Oct 31 03:24:35.795252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:35.795145 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.246464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.246350 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.391030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.390990 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.523429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.523332 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.668193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.668153 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:36.681803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.680370 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:24:36.681803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.680671 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" containerID="cri-o://b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf" gracePeriod=30 Oct 31 03:24:36.697878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.693265 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:24:36.698240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.698188 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" containerID="cri-o://8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8" gracePeriod=30 Oct 31 03:24:36.762989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.762945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:36.900362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:36.900326 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:37.207054 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:37.206945 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:37.406337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:37.406300 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:37.519206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:37.519120 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:24:37.960169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:37.960123 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:38.210731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:38.210650 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:38.949021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:38.948986 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:39.641287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:39.639379 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:40.480731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:40.480689 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:41.024608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.024574 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:24:41.053294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.053223 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:24:41.096748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.096718 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:24:41.182036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.181996 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.182233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182055 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.182233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182115 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.182233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182157 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.182233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182185 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.182555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182239 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.182620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182566 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.182620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182592 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.182892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182804 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.183012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182903 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.183159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183132 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.183300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.182935 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.183462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183426 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.183619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183605 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.183725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183706 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.183802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183759 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.183802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183784 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.184033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.183992 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.184381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.184339 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.184695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.184675 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.185255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.185236 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.186155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.185845 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.186572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.186297 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.186972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.185450 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.186972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.186093 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.186972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.186529 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.186972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.186796 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.186972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.186947 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "kube-api-access-jx9qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.187285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187074 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.187381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187361 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.187470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187402 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187475 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187510 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187592 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187358 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187630 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187595 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187659 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "kube-api-access-f5xhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187704 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187783 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.187823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187787 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.188431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.187916 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.188658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188619 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\") pod \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\" (UID: \"bd3ab0f9-98e5-42b6-95fb-86317586e85c\") " Oct 31 03:24:41.188770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188672 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.188770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188706 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\") pod \"75a20ec2-690a-4360-9287-a2f8f85f09b4\" (UID: \"75a20ec2-690a-4360-9287-a2f8f85f09b4\") " Oct 31 03:24:41.188770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188752 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\") pod \"137b2f24-7930-414e-acec-df30d6a1f713\" (UID: \"137b2f24-7930-414e-acec-df30d6a1f713\") " Oct 31 03:24:41.188949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188866 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.189354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.188800 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:41.189875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.189846 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:41.191860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.191825 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "kube-api-access-47c6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.191996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.191935 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:41.192244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192221 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token" pod "137b2f24-7930-414e-acec-df30d6a1f713" (UID: "137b2f24-7930-414e-acec-df30d6a1f713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.192535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192495 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token" pod "75a20ec2-690a-4360-9287-a2f8f85f09b4" (UID: "75a20ec2-690a-4360-9287-a2f8f85f09b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.192535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192523 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192547 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192564 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192587 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xhx\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-kube-api-access-f5xhx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192602 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192616 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192630 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192651 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192666 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192680 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a20ec2-690a-4360-9287-a2f8f85f09b4-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192694 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b2f24-7930-414e-acec-df30d6a1f713-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.192716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192713 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192726 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47c6j\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-kube-api-access-47c6j\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192741 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192757 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192775 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192789 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx9qg\" (UniqueName: \"kubernetes.io/projected/137b2f24-7930-414e-acec-df30d6a1f713-kube-api-access-jx9qg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192803 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192821 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192837 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b2f24-7930-414e-acec-df30d6a1f713-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192852 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192866 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192883 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192898 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b2f24-7930-414e-acec-df30d6a1f713-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.193334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.192911 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3ab0f9-98e5-42b6-95fb-86317586e85c-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.194563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.194534 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token" pod "bd3ab0f9-98e5-42b6-95fb-86317586e85c" (UID: "bd3ab0f9-98e5-42b6-95fb-86317586e85c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:41.238260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.238216 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:41.294823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.294734 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a20ec2-690a-4360-9287-a2f8f85f09b4-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.294823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.294769 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3ab0f9-98e5-42b6-95fb-86317586e85c-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:41.326041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.325997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:41.749838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.749804 4556 generic.go:356] "Generic (PLEG): container finished" podID="137b2f24-7930-414e-acec-df30d6a1f713" containerID="ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff" exitCode=0 Oct 31 03:24:41.750317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.749881 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerDied","Data":"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff"} Oct 31 03:24:41.750317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.749938 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" Oct 31 03:24:41.750317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.749957 4556 scope.go:122] "RemoveContainer" containerID="ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff" Oct 31 03:24:41.750317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.749945 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9" event={"ID":"137b2f24-7930-414e-acec-df30d6a1f713","Type":"ContainerDied","Data":"70fd71955082392d41d1aef4d3879e03da468114ce9e1cb8ac9bc7bab840b38e"} Oct 31 03:24:41.751589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.751562 4556 generic.go:356] "Generic (PLEG): container finished" podID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerID="c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976" exitCode=0 Oct 31 03:24:41.751710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.751616 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" Oct 31 03:24:41.751710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.751639 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" event={"ID":"bd3ab0f9-98e5-42b6-95fb-86317586e85c","Type":"ContainerDied","Data":"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976"} Oct 31 03:24:41.751710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.751676 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg" event={"ID":"bd3ab0f9-98e5-42b6-95fb-86317586e85c","Type":"ContainerDied","Data":"6426476196f5c381ba13ac2f91f1b18df52ce3e99c55bf64a759221073ac987d"} Oct 31 03:24:41.753275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.753247 4556 generic.go:356] "Generic (PLEG): container finished" podID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerID="edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6" exitCode=0 Oct 31 03:24:41.753398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.753289 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" event={"ID":"75a20ec2-690a-4360-9287-a2f8f85f09b4","Type":"ContainerDied","Data":"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6"} Oct 31 03:24:41.753398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.753310 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" Oct 31 03:24:41.753398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.753337 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq" event={"ID":"75a20ec2-690a-4360-9287-a2f8f85f09b4","Type":"ContainerDied","Data":"8d2d8785a392de23b46d33b6f12f4bab3ce31dea2a1fb5e5e7a82217ed374e0e"} Oct 31 03:24:41.839186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.839164 4556 scope.go:122] "RemoveContainer" containerID="a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8" Oct 31 03:24:41.853745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.853720 4556 scope.go:122] "RemoveContainer" containerID="ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff" Oct 31 03:24:41.854275 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:41.854245 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff\": container with ID starting with ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff not found: ID does not exist" containerID="ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff" Oct 31 03:24:41.854337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.854290 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff"} err="failed to get container status \"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff\": rpc error: code = NotFound desc = could not find container \"ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff\": container with ID starting with ac106271faa6682af70dcbbf72ddd36625e73166589e742303a8c06ab68d8dff not found: ID does not exist" Oct 31 03:24:41.854337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.854327 4556 scope.go:122] "RemoveContainer" containerID="a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8" Oct 31 03:24:41.854847 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:41.854799 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8\": container with ID starting with a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8 not found: ID does not exist" containerID="a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8" Oct 31 03:24:41.854983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.854855 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8"} err="failed to get container status \"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8\": rpc error: code = NotFound desc = could not find container \"a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8\": container with ID starting with a06ed72637d2675815f6f3ae2dbec3c12a38664fb8686cc4837ede196e5fa6d8 not found: ID does not exist" Oct 31 03:24:41.854983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.854876 4556 scope.go:122] "RemoveContainer" containerID="c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976" Oct 31 03:24:41.857255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.857233 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:24:41.859937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.859909 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-rlnpq"] Oct 31 03:24:41.875148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.875115 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:24:41.875381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.875366 4556 scope.go:122] "RemoveContainer" containerID="c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976" Oct 31 03:24:41.875802 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:41.875786 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976\": container with ID starting with c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976 not found: ID does not exist" containerID="c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976" Oct 31 03:24:41.875862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.875811 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976"} err="failed to get container status \"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976\": rpc error: code = NotFound desc = could not find container \"c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976\": container with ID starting with c5786075600eeb7dc94088e608c646ac498ffec08712f928192195ec98149976 not found: ID does not exist" Oct 31 03:24:41.875862 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.875828 4556 scope.go:122] "RemoveContainer" containerID="edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6" Oct 31 03:24:41.882377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.882327 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-44dd9"] Oct 31 03:24:41.888920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.888876 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:24:41.890896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.890844 4556 scope.go:122] "RemoveContainer" containerID="edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6" Oct 31 03:24:41.891521 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:41.891389 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6\": container with ID starting with edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6 not found: ID does not exist" containerID="edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6" Oct 31 03:24:41.891678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.891644 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6"} err="failed to get container status \"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6\": rpc error: code = NotFound desc = could not find container \"edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6\": container with ID starting with edb7570a671b101f6a3fcf6af8729f438e57fd6b4704781f9ff788efe1d9fdd6 not found: ID does not exist" Oct 31 03:24:41.896155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:41.896132 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-62bf00-openshift-default-858f469555-tsrbg"] Oct 31 03:24:42.021798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.021760 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:24:42.060073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.060034 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:24:42.103892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.103840 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.103956 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.103989 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104034 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104166 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104209 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.104276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104236 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104302 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.104464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104367 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104462 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.104775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104500 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.104775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104580 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.104775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104540 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\") pod \"3e66071d-8b8e-4713-b328-5b86e2510379\" (UID: \"3e66071d-8b8e-4713-b328-5b86e2510379\") " Oct 31 03:24:42.105011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.104775 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.105315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.105288 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.105454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.105320 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.105454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.105340 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.105454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.105355 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.105454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.105368 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e66071d-8b8e-4713-b328-5b86e2510379-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.106936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.106905 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.107206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.107185 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "kube-api-access-jrfqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.107271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.107253 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.107315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.107299 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo" pod "3e66071d-8b8e-4713-b328-5b86e2510379" (UID: "3e66071d-8b8e-4713-b328-5b86e2510379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.136712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.136622 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:42.206386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206300 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206393 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206472 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206623 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206657 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206687 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206717 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.206903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206779 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.206903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206788 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.207055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206928 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\") pod \"c65a5e49-14c2-4907-b295-6ccc473195b5\" (UID: \"c65a5e49-14c2-4907-b295-6ccc473195b5\") " Oct 31 03:24:42.207055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206936 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.207055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206978 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.207055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.206976 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:42.207284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207264 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.207587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207567 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207586 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e66071d-8b8e-4713-b328-5b86e2510379-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207595 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207606 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207615 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207623 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65a5e49-14c2-4907-b295-6ccc473195b5-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207632 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207641 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e66071d-8b8e-4713-b328-5b86e2510379-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.207759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.207651 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfqt\" (UniqueName: \"kubernetes.io/projected/3e66071d-8b8e-4713-b328-5b86e2510379-kube-api-access-jrfqt\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.209074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.209043 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:42.209305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.209218 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:42.209305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.209272 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "kube-api-access-bqs8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.209665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.209643 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token" pod "c65a5e49-14c2-4907-b295-6ccc473195b5" (UID: "c65a5e49-14c2-4907-b295-6ccc473195b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:42.214486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.214463 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:24:42.225038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.225005 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:24:42.259745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.259707 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137b2f24-7930-414e-acec-df30d6a1f713" path="/var/lib/kubelet/pods/137b2f24-7930-414e-acec-df30d6a1f713/volumes" Oct 31 03:24:42.260341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.260317 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" path="/var/lib/kubelet/pods/75a20ec2-690a-4360-9287-a2f8f85f09b4/volumes" Oct 31 03:24:42.260923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.260902 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" path="/var/lib/kubelet/pods/bd3ab0f9-98e5-42b6-95fb-86317586e85c/volumes" Oct 31 03:24:42.308681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.308639 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.308681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.308675 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.308681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.308687 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqs8p\" (UniqueName: \"kubernetes.io/projected/c65a5e49-14c2-4907-b295-6ccc473195b5-kube-api-access-bqs8p\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.308928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.308699 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65a5e49-14c2-4907-b295-6ccc473195b5-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:42.353581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.353546 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:42.763835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.763695 4556 generic.go:356] "Generic (PLEG): container finished" podID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerID="8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8" exitCode=0 Oct 31 03:24:42.763835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.763751 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" Oct 31 03:24:42.763835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.763771 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" event={"ID":"c65a5e49-14c2-4907-b295-6ccc473195b5","Type":"ContainerDied","Data":"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8"} Oct 31 03:24:42.764438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.763837 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84" event={"ID":"c65a5e49-14c2-4907-b295-6ccc473195b5","Type":"ContainerDied","Data":"ce369108ff60121c0aac1ba16a6c9858efcba40625da2e699a34c8da6455efbd"} Oct 31 03:24:42.764438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.763872 4556 scope.go:122] "RemoveContainer" containerID="8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8" Oct 31 03:24:42.765652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.765626 4556 generic.go:356] "Generic (PLEG): container finished" podID="3e66071d-8b8e-4713-b328-5b86e2510379" containerID="b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf" exitCode=0 Oct 31 03:24:42.765765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.765686 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" Oct 31 03:24:42.765765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.765743 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" event={"ID":"3e66071d-8b8e-4713-b328-5b86e2510379","Type":"ContainerDied","Data":"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf"} Oct 31 03:24:42.765897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.765784 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5" event={"ID":"3e66071d-8b8e-4713-b328-5b86e2510379","Type":"ContainerDied","Data":"faddc6dd48348b400f5beb5cf1c98410c09d553f49bc6a8c56205f99ca45aaa7"} Oct 31 03:24:42.779062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.779043 4556 scope.go:122] "RemoveContainer" containerID="8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8" Oct 31 03:24:42.779390 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:42.779365 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8\": container with ID starting with 8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8 not found: ID does not exist" containerID="8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8" Oct 31 03:24:42.779504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.779402 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8"} err="failed to get container status \"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8\": rpc error: code = NotFound desc = could not find container \"8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8\": container with ID starting with 8eca7c86fa2ac511d219ecc9615dd51ffc887465fa3c640bb5a04b78431b8cd8 not found: ID does not exist" Oct 31 03:24:42.779504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.779437 4556 scope.go:122] "RemoveContainer" containerID="b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf" Oct 31 03:24:42.793963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.793931 4556 scope.go:122] "RemoveContainer" containerID="b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf" Oct 31 03:24:42.794339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.794315 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:24:42.794339 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:42.794331 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf\": container with ID starting with b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf not found: ID does not exist" containerID="b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf" Oct 31 03:24:42.794526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.794362 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf"} err="failed to get container status \"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf\": rpc error: code = NotFound desc = could not find container \"b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf\": container with ID starting with b3c0ba005b4f3ca13c5ce77948d1c5deb13bc8ae17addb8c2fa2a4684b78c3bf not found: ID does not exist" Oct 31 03:24:42.797004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.796981 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-79sc5"] Oct 31 03:24:42.808019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.807984 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:24:42.812089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.812059 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7269ae-openshift-default-759fbbf-dlr84"] Oct 31 03:24:42.956574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:42.955905 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:43.793840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:43.793802 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:44.258259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:44.258222 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" path="/var/lib/kubelet/pods/3e66071d-8b8e-4713-b328-5b86e2510379/volumes" Oct 31 03:24:44.258788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:44.258770 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" path="/var/lib/kubelet/pods/c65a5e49-14c2-4907-b295-6ccc473195b5/volumes" Oct 31 03:24:44.518691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:44.517821 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:44.659796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:44.659757 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:24:44.660028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:44.659990 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" containerID="cri-o://98a8959ce7a195b422e71ffdb06ad395ead49b714dd6a11cae6cbb850605c189" gracePeriod=30 Oct 31 03:24:45.286933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:45.286896 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:45.937351 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:45.937311 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:24:46.038453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.038398 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:46.437298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.437265 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:46.584450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.581587 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:46.665340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.665306 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:46.757561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.757470 4556 patch_prober.go:29] interesting pod/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:46.757561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.757545 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:46.905213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:46.905176 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:47.511398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.511360 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:47.531274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.528362 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:24:47.544521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.544428 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:24:47.619228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.608605 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:24:47.646881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.646844 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:47.668642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.668613 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:24:47.687799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.687361 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\" (UniqueName: \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\") pod \"dedc3cc2-fb07-49bb-a553-3283341edeaf\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " Oct 31 03:24:47.687799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.687497 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\") pod \"dedc3cc2-fb07-49bb-a553-3283341edeaf\" (UID: \"dedc3cc2-fb07-49bb-a553-3283341edeaf\") " Oct 31 03:24:47.687799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.687532 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\") pod \"8ba3de21-6f99-4686-838c-4b35ea81602c\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " Oct 31 03:24:47.687799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.687566 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\" (UniqueName: \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\") pod \"8ba3de21-6f99-4686-838c-4b35ea81602c\" (UID: \"8ba3de21-6f99-4686-838c-4b35ea81602c\") " Oct 31 03:24:47.688988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.688910 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume" pod "8ba3de21-6f99-4686-838c-4b35ea81602c" (UID: "8ba3de21-6f99-4686-838c-4b35ea81602c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:47.689156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.689005 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume" pod "dedc3cc2-fb07-49bb-a553-3283341edeaf" (UID: "dedc3cc2-fb07-49bb-a553-3283341edeaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:47.693843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.693802 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc" pod "dedc3cc2-fb07-49bb-a553-3283341edeaf" (UID: "dedc3cc2-fb07-49bb-a553-3283341edeaf"). InnerVolumeSpecName "kube-api-access-mrnlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:47.694262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.694233 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h" pod "8ba3de21-6f99-4686-838c-4b35ea81602c" (UID: "8ba3de21-6f99-4686-838c-4b35ea81602c"). InnerVolumeSpecName "kube-api-access-cww7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:47.773276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.773227 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:47.792162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.792119 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrnlc\" (UniqueName: \"kubernetes.io/projected/dedc3cc2-fb07-49bb-a553-3283341edeaf-kube-api-access-mrnlc\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:47.792162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.792157 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dedc3cc2-fb07-49bb-a553-3283341edeaf-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:47.792453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.792174 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba3de21-6f99-4686-838c-4b35ea81602c-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:47.792453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.792188 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cww7h\" (UniqueName: \"kubernetes.io/projected/8ba3de21-6f99-4686-838c-4b35ea81602c-kube-api-access-cww7h\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:47.808745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.808160 4556 generic.go:356] "Generic (PLEG): container finished" podID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerID="0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16" exitCode=0 Oct 31 03:24:47.808745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.808308 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" Oct 31 03:24:47.808745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.808300 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" event={"ID":"8ba3de21-6f99-4686-838c-4b35ea81602c","Type":"ContainerDied","Data":"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16"} Oct 31 03:24:47.808745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.808494 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w" event={"ID":"8ba3de21-6f99-4686-838c-4b35ea81602c","Type":"ContainerDied","Data":"e3bf3b79bb29c9d516effccb4d873988e3af8d6daff67ea8560e4c064810d2ec"} Oct 31 03:24:47.808745 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.808530 4556 scope.go:122] "RemoveContainer" containerID="0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16" Oct 31 03:24:47.833454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.827825 4556 generic.go:356] "Generic (PLEG): container finished" podID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerID="a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8" exitCode=0 Oct 31 03:24:47.833454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.827907 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" event={"ID":"dedc3cc2-fb07-49bb-a553-3283341edeaf","Type":"ContainerDied","Data":"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8"} Oct 31 03:24:47.833454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.827962 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" event={"ID":"dedc3cc2-fb07-49bb-a553-3283341edeaf","Type":"ContainerDied","Data":"4746679d7829d20a15f436232f9eb815c537b5945dcbe991459e68a6b9685897"} Oct 31 03:24:47.833454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.828060 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7" Oct 31 03:24:47.890442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.874854 4556 scope.go:122] "RemoveContainer" containerID="0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16" Oct 31 03:24:47.890442 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:47.879619 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16\": container with ID starting with 0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16 not found: ID does not exist" containerID="0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16" Oct 31 03:24:47.890442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.879664 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16"} err="failed to get container status \"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16\": rpc error: code = NotFound desc = could not find container \"0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16\": container with ID starting with 0fb7e746dfd38ce80d30f59628316d7bed976f40a3727a30d265ec5c2398fd16 not found: ID does not exist" Oct 31 03:24:47.890442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.879703 4556 scope.go:122] "RemoveContainer" containerID="a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8" Oct 31 03:24:47.910694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.901494 4556 scope.go:122] "RemoveContainer" containerID="a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8" Oct 31 03:24:47.910694 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:47.901906 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8\": container with ID starting with a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8 not found: ID does not exist" containerID="a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8" Oct 31 03:24:47.910694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.901949 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8"} err="failed to get container status \"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8\": rpc error: code = NotFound desc = could not find container \"a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8\": container with ID starting with a87e9f5d1772339cb012484e92a01453512a926674fc0d0beadc5604b0b9c2c8 not found: ID does not exist" Oct 31 03:24:47.910694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.902822 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:47.927476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.916634 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:24:47.927476 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.923068 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62bf00-5b6977886f-sjt9w"] Oct 31 03:24:47.928751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.928722 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:24:47.932346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:47.932295 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62bf00-6c4c684655-vvzh7"] Oct 31 03:24:48.034651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:48.034615 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:48.258807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:48.258775 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" path="/var/lib/kubelet/pods/8ba3de21-6f99-4686-838c-4b35ea81602c/volumes" Oct 31 03:24:48.259319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:48.259301 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" path="/var/lib/kubelet/pods/dedc3cc2-fb07-49bb-a553-3283341edeaf/volumes" Oct 31 03:24:48.392692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:48.392661 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:48.934789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:48.934754 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:49.730038 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:49.728788 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:49.852256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:49.852214 4556 generic.go:356] "Generic (PLEG): container finished" podID="27f16360-e558-4069-99ff-a4cfd951257f" containerID="98a8959ce7a195b422e71ffdb06ad395ead49b714dd6a11cae6cbb850605c189" exitCode=0 Oct 31 03:24:49.852459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:49.852273 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerDied","Data":"98a8959ce7a195b422e71ffdb06ad395ead49b714dd6a11cae6cbb850605c189"} Oct 31 03:24:49.852459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:49.852395 4556 scope.go:122] "RemoveContainer" containerID="1080c06b96636257f917c67329925c33db7746d873b7f51e3f66625120c44817" Oct 31 03:24:49.929994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:49.929917 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:24:50.023128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023083 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023175 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023232 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023318 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023386 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023431 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023443 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023514 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023542 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.023697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023591 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023740 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\") pod \"27f16360-e558-4069-99ff-a4cfd951257f\" (UID: \"27f16360-e558-4069-99ff-a4cfd951257f\") " Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.023837 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.024306 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.024361 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.024386 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.024401 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f16360-e558-4069-99ff-a4cfd951257f-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.024992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.024530 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:50.026995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.026969 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "kube-api-access-dvbtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:50.027099 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.027020 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:50.027598 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.027579 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:24:50.027650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.027595 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy" pod "27f16360-e558-4069-99ff-a4cfd951257f" (UID: "27f16360-e558-4069-99ff-a4cfd951257f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:50.126071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126023 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.126071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126061 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f16360-e558-4069-99ff-a4cfd951257f-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.126071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126075 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbtx\" (UniqueName: \"kubernetes.io/projected/27f16360-e558-4069-99ff-a4cfd951257f-kube-api-access-dvbtx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.126338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126086 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.126338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126095 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.126338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.126104 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f16360-e558-4069-99ff-a4cfd951257f-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:50.626863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.626715 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:50.860229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.860196 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" Oct 31 03:24:50.860229 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.860212 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26" event={"ID":"27f16360-e558-4069-99ff-a4cfd951257f","Type":"ContainerDied","Data":"74de6e23dfb65afa1f19d0b5286e03e72636f5e3e80932310ef166da931462b5"} Oct 31 03:24:50.860532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.860274 4556 scope.go:122] "RemoveContainer" containerID="98a8959ce7a195b422e71ffdb06ad395ead49b714dd6a11cae6cbb850605c189" Oct 31 03:24:50.879946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.879870 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:24:50.887442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:50.887400 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-thc26"] Oct 31 03:24:51.237858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.237816 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:51.543529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.543434 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:51.655521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.655479 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:24:51.766437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.755017 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:24:51.852750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.852655 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\") pod \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " Oct 31 03:24:51.852927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.852831 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\" (UniqueName: \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\") pod \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\" (UID: \"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f\") " Oct 31 03:24:51.853063 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.853038 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume" pod "3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" (UID: "3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:24:51.853199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.853182 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:51.855327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.855297 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl" pod "3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" (UID: "3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f"). InnerVolumeSpecName "kube-api-access-ttzgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:51.868692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.868657 4556 generic.go:356] "Generic (PLEG): container finished" podID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerID="6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72" exitCode=0 Oct 31 03:24:51.868846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.868710 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" Oct 31 03:24:51.868846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.868735 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" event={"ID":"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f","Type":"ContainerDied","Data":"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72"} Oct 31 03:24:51.868846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.868794 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59" event={"ID":"3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f","Type":"ContainerDied","Data":"f371df8a6069c568fc328d4dab2ac54093aa7f5a69f814ec5b7e344fff350057"} Oct 31 03:24:51.868846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.868807 4556 scope.go:122] "RemoveContainer" containerID="6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72" Oct 31 03:24:51.878899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.878873 4556 scope.go:122] "RemoveContainer" containerID="6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72" Oct 31 03:24:51.879221 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:51.879201 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72\": container with ID starting with 6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72 not found: ID does not exist" containerID="6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72" Oct 31 03:24:51.879274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.879230 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72"} err="failed to get container status \"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72\": rpc error: code = NotFound desc = could not find container \"6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72\": container with ID starting with 6328089f2a2dbfeeda423137c7971fad569fcfeee1c476b637e7db68e557bf72 not found: ID does not exist" Oct 31 03:24:51.891006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.890969 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:24:51.895778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.895745 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7269ae-56d579cf79-pmg59"] Oct 31 03:24:51.954630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:51.954583 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzgl\" (UniqueName: \"kubernetes.io/projected/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f-kube-api-access-ttzgl\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:52.266045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.266013 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f16360-e558-4069-99ff-a4cfd951257f" path="/var/lib/kubelet/pods/27f16360-e558-4069-99ff-a4cfd951257f/volumes" Oct 31 03:24:52.266610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.266592 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" path="/var/lib/kubelet/pods/3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f/volumes" Oct 31 03:24:52.435397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.435352 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:52.529796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.529693 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" containerID="cri-o://5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e" gracePeriod=25 Oct 31 03:24:52.545323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.545269 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" containerID="cri-o://1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc" gracePeriod=25 Oct 31 03:24:52.905217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:52.905178 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:24:53.393500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:53.393390 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:53.898021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:53.897975 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:54.055357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:54.055320 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:54.745193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:54.745151 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:55.426498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:55.426205 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:55.992325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:55.992291 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:56.118757 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:56.118707 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:56.250160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:56.250024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:56.266689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:56.266642 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:24:56.266980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:56.266950 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" containerID="cri-o://54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312" gracePeriod=30 Oct 31 03:24:56.411653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:56.411610 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:57.003612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.003572 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:57.240738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.240695 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" containerID="cri-o://3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d" gracePeriod=45 Oct 31 03:24:57.257071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.256987 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" containerID="cri-o://30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519" gracePeriod=45 Oct 31 03:24:57.386578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.386539 4556 patch_prober.go:29] interesting pod/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:24:57.386769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.386620 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:24:57.461209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.460859 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 31 03:24:57.578665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.578637 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:24:57.588517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.585585 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:24:57.631118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631069 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.631318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631196 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.631318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631239 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.631318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631271 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.631565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631375 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.631565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631447 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\" (UniqueName: \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.631565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631472 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.631565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631500 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.631793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631557 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.631793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631602 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.631793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631699 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.631793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631761 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.632015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631795 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.632015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631846 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\" (UniqueName: \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\") pod \"4b702eeb-fa9e-460b-8c81-1feeea666d69\" (UID: \"4b702eeb-fa9e-460b-8c81-1feeea666d69\") " Oct 31 03:24:57.632015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.631906 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.632015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632007 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.632224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632038 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\") pod \"70af976b-5785-4f55-ba46-c652c94a56c0\" (UID: \"70af976b-5785-4f55-ba46-c652c94a56c0\") " Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632347 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632821 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632842 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632860 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632874 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632886 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.632935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.632900 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.636794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.636762 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "kube-api-access-qlp2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:57.637617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.637591 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "kube-api-access-w2j9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:57.637814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.637796 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.637943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.637807 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs" pod "70af976b-5785-4f55-ba46-c652c94a56c0" (UID: "70af976b-5785-4f55-ba46-c652c94a56c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:57.638447 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.638397 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:24:57.638723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.638692 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs" pod "4b702eeb-fa9e-460b-8c81-1feeea666d69" (UID: "4b702eeb-fa9e-460b-8c81-1feeea666d69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:57.734427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734379 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70af976b-5785-4f55-ba46-c652c94a56c0-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.734427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734426 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b702eeb-fa9e-460b-8c81-1feeea666d69-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.734640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734444 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlp2w\" (UniqueName: \"kubernetes.io/projected/70af976b-5785-4f55-ba46-c652c94a56c0-kube-api-access-qlp2w\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.734640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734454 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b702eeb-fa9e-460b-8c81-1feeea666d69-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.734640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734464 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j9n\" (UniqueName: \"kubernetes.io/projected/4b702eeb-fa9e-460b-8c81-1feeea666d69-kube-api-access-w2j9n\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.734640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.734471 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70af976b-5785-4f55-ba46-c652c94a56c0-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:57.840098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.839996 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:57.907170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.907136 4556 generic.go:356] "Generic (PLEG): container finished" podID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerID="3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d" exitCode=0 Oct 31 03:24:57.907397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.907181 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" Oct 31 03:24:57.907397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.907227 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" event={"ID":"4b702eeb-fa9e-460b-8c81-1feeea666d69","Type":"ContainerDied","Data":"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d"} Oct 31 03:24:57.907397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.907269 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh" event={"ID":"4b702eeb-fa9e-460b-8c81-1feeea666d69","Type":"ContainerDied","Data":"3bf535ab9816302158788c1dd5341c3796f40a192f81e53c5cdf0d0b137c8f9a"} Oct 31 03:24:57.907397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.907296 4556 scope.go:122] "RemoveContainer" containerID="3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d" Oct 31 03:24:57.908867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.908835 4556 generic.go:356] "Generic (PLEG): container finished" podID="70af976b-5785-4f55-ba46-c652c94a56c0" containerID="30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519" exitCode=0 Oct 31 03:24:57.909010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.908866 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" event={"ID":"70af976b-5785-4f55-ba46-c652c94a56c0","Type":"ContainerDied","Data":"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519"} Oct 31 03:24:57.909010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.908900 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" event={"ID":"70af976b-5785-4f55-ba46-c652c94a56c0","Type":"ContainerDied","Data":"41334d13b93169e695a99a1584c16316357c04bcaae189a029f54efdf9c05fe4"} Oct 31 03:24:57.909010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.908899 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5" Oct 31 03:24:57.919542 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.919514 4556 scope.go:122] "RemoveContainer" containerID="3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d" Oct 31 03:24:57.919945 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:57.919912 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d\": container with ID starting with 3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d not found: ID does not exist" containerID="3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d" Oct 31 03:24:57.919993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.919941 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d"} err="failed to get container status \"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d\": rpc error: code = NotFound desc = could not find container \"3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d\": container with ID starting with 3470dccbe2185aed326915af36d7ea2d8697801a4afafc6bf3f6832c4fd2576d not found: ID does not exist" Oct 31 03:24:57.919993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.919960 4556 scope.go:122] "RemoveContainer" containerID="30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519" Oct 31 03:24:57.929826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.929801 4556 scope.go:122] "RemoveContainer" containerID="30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519" Oct 31 03:24:57.930167 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:24:57.930143 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519\": container with ID starting with 30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519 not found: ID does not exist" containerID="30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519" Oct 31 03:24:57.930218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.930177 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519"} err="failed to get container status \"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519\": rpc error: code = NotFound desc = could not find container \"30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519\": container with ID starting with 30155b07ef5ea4ebe433fe2a85ecac4bb3bed64dfa834055e73a4f823b4b5519 not found: ID does not exist" Oct 31 03:24:57.934690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.934661 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:24:57.941755 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.941715 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/simulator-w1-924ee5-kserve-678b85cb94-xzlf5"] Oct 31 03:24:57.954139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.954084 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:24:57.958786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:57.958742 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-924ee5/premium-w1-924ee5-kserve-6d7988bd59-999rh"] Oct 31 03:24:58.257641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:58.257602 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" path="/var/lib/kubelet/pods/4b702eeb-fa9e-460b-8c81-1feeea666d69/volumes" Oct 31 03:24:58.258112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:58.258069 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" path="/var/lib/kubelet/pods/70af976b-5785-4f55-ba46-c652c94a56c0/volumes" Oct 31 03:24:58.831265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:58.827543 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:59.528886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:24:59.528840 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:00.221953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:00.221903 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:01.020699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.020665 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:01.560601 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.560529 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:25:01.691193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691154 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691207 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691244 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691324 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691372 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691435 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691466 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691494 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691520 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\") pod \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\" (UID: \"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9\") " Oct 31 03:25:01.691866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691549 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.691866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691812 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.692034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691929 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.692034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.691955 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:01.692143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.692064 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.692143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.692077 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.692143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.692088 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.692143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.692109 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.692368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.692262 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.694631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.694592 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "kube-api-access-mk4wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:01.694772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.694643 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:01.695043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.695020 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:01.695545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.695519 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo" pod "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" (UID: "0ee7dd0d-24dc-457c-ad21-38a84b07f1b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:01.793581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.793536 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.793581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.793578 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.793815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.793594 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4wn\" (UniqueName: \"kubernetes.io/projected/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-kube-api-access-mk4wn\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.793815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.793611 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.793815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.793626 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:01.830860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.830742 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:01.899793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.899749 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:25:01.936115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.936076 4556 generic.go:356] "Generic (PLEG): container finished" podID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerID="54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312" exitCode=0 Oct 31 03:25:01.936115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.936118 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" Oct 31 03:25:01.936402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.936146 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" event={"ID":"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9","Type":"ContainerDied","Data":"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312"} Oct 31 03:25:01.936402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.936205 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj" event={"ID":"0ee7dd0d-24dc-457c-ad21-38a84b07f1b9","Type":"ContainerDied","Data":"b45574307543002443143c28ccfcb53f44201a295d5d872e3bef03014c26b2ce"} Oct 31 03:25:01.936402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.936230 4556 scope.go:122] "RemoveContainer" containerID="54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312" Oct 31 03:25:01.946970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.946944 4556 scope.go:122] "RemoveContainer" containerID="54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312" Oct 31 03:25:01.947312 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:01.947289 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312\": container with ID starting with 54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312 not found: ID does not exist" containerID="54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312" Oct 31 03:25:01.947389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.947323 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312"} err="failed to get container status \"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312\": rpc error: code = NotFound desc = could not find container \"54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312\": container with ID starting with 54204e196ee760ea622ca6572039596a2bf1f1a04d827a8b26b9835966bee312 not found: ID does not exist" Oct 31 03:25:01.959859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.959791 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:25:01.962378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:01.962345 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b79c13-openshift-default-78b6fb5955-wjdhj"] Oct 31 03:25:02.257104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:02.257070 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" path="/var/lib/kubelet/pods/0ee7dd0d-24dc-457c-ad21-38a84b07f1b9/volumes" Oct 31 03:25:02.609845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:02.609730 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:02.905581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:02.905537 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:25:03.329074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:03.328946 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:04.090349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:04.090307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:04.764505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:04.764473 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:05.449461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:05.438715 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.127668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:06.127407 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.700936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:06.700902 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.860012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:06.859978 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:06.988974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:06.988886 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.134683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:07.134638 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.282491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:07.271974 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.535085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:07.534994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:07.897791 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:07.897702 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:08.730047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:08.729979 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:09.438270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:09.438236 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:09.456705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:09.456665 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:25:09.579794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:09.579755 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.136788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:10.136751 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.288250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:10.288213 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.433284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:10.433245 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:10.570838 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:10.570803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:11.014357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:11.014323 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:11.447257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:11.447217 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:11.900593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:11.900553 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:25:12.139697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.139628 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:12.821160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.821133 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:25:12.851161 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.851120 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\") pod \"4b869789-ca41-4036-9c44-d1fbe68481df\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " Oct 31 03:25:12.851925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.851360 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\" (UniqueName: \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\") pod \"4b869789-ca41-4036-9c44-d1fbe68481df\" (UID: \"4b869789-ca41-4036-9c44-d1fbe68481df\") " Oct 31 03:25:12.851925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.851591 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume" pod "4b869789-ca41-4036-9c44-d1fbe68481df" (UID: "4b869789-ca41-4036-9c44-d1fbe68481df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:12.851925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.851888 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b869789-ca41-4036-9c44-d1fbe68481df-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:12.853931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.853908 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg" pod "4b869789-ca41-4036-9c44-d1fbe68481df" (UID: "4b869789-ca41-4036-9c44-d1fbe68481df"). InnerVolumeSpecName "kube-api-access-wb2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:12.862269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.862242 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:25:12.929360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.929324 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:12.967441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.966946 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\") pod \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " Oct 31 03:25:12.967441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.967178 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\") pod \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\" (UID: \"952f64f5-b2ba-46d9-b45a-c5d3ab037827\") " Oct 31 03:25:12.967731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.967609 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2lg\" (UniqueName: \"kubernetes.io/projected/4b869789-ca41-4036-9c44-d1fbe68481df-kube-api-access-wb2lg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:12.985012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.968843 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume" pod "952f64f5-b2ba-46d9-b45a-c5d3ab037827" (UID: "952f64f5-b2ba-46d9-b45a-c5d3ab037827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:12.996630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:12.996573 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2" pod "952f64f5-b2ba-46d9-b45a-c5d3ab037827" (UID: "952f64f5-b2ba-46d9-b45a-c5d3ab037827"). InnerVolumeSpecName "kube-api-access-b5pf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:13.013982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.010312 4556 generic.go:356] "Generic (PLEG): container finished" podID="4b869789-ca41-4036-9c44-d1fbe68481df" containerID="5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e" exitCode=0 Oct 31 03:25:13.013982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.010388 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" Oct 31 03:25:13.013982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.010494 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" event={"ID":"4b869789-ca41-4036-9c44-d1fbe68481df","Type":"ContainerDied","Data":"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e"} Oct 31 03:25:13.013982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.010540 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8" event={"ID":"4b869789-ca41-4036-9c44-d1fbe68481df","Type":"ContainerDied","Data":"fb5f76f1f14eabae96fb99472b37672493c4f746881c48967ebd1e0fbe85c213"} Oct 31 03:25:13.013982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.010570 4556 scope.go:122] "RemoveContainer" containerID="5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e" Oct 31 03:25:13.018545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.015146 4556 generic.go:356] "Generic (PLEG): container finished" podID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerID="1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc" exitCode=0 Oct 31 03:25:13.018545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.015294 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" event={"ID":"952f64f5-b2ba-46d9-b45a-c5d3ab037827","Type":"ContainerDied","Data":"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc"} Oct 31 03:25:13.018545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.015346 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" event={"ID":"952f64f5-b2ba-46d9-b45a-c5d3ab037827","Type":"ContainerDied","Data":"b705c6f4e8eb0cf9a239cf7faea63f405e2bfa7572569c8038cadf2a54a6dbfe"} Oct 31 03:25:13.018545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.015447 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq" Oct 31 03:25:13.034729 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.034703 4556 scope.go:122] "RemoveContainer" containerID="5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e" Oct 31 03:25:13.035563 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:13.035312 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e\": container with ID starting with 5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e not found: ID does not exist" containerID="5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e" Oct 31 03:25:13.035563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.035352 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e"} err="failed to get container status \"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e\": rpc error: code = NotFound desc = could not find container \"5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e\": container with ID starting with 5ef0a0356982ed29890999bc478d391847d086d09fa5c6ad46867af669493e4e not found: ID does not exist" Oct 31 03:25:13.035563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.035371 4556 scope.go:122] "RemoveContainer" containerID="1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc" Oct 31 03:25:13.052871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.052819 4556 scope.go:122] "RemoveContainer" containerID="1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc" Oct 31 03:25:13.053319 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:13.053291 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc\": container with ID starting with 1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc not found: ID does not exist" containerID="1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc" Oct 31 03:25:13.053406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.053332 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc"} err="failed to get container status \"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc\": rpc error: code = NotFound desc = could not find container \"1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc\": container with ID starting with 1fb18f779327e5e6b5dabdc5c14b8dee8efa1efce44f42536c572e0d4f1924dc not found: ID does not exist" Oct 31 03:25:13.054182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.054158 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:25:13.060127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.060098 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-924ee5-f8db9b8b7-m5mgq"] Oct 31 03:25:13.068370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.068337 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5pf2\" (UniqueName: \"kubernetes.io/projected/952f64f5-b2ba-46d9-b45a-c5d3ab037827-kube-api-access-b5pf2\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:13.068539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.068374 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/952f64f5-b2ba-46d9-b45a-c5d3ab037827-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:13.068539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.068447 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:25:13.074671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.074638 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-924ee5-65c97c4d6f-2f9j8"] Oct 31 03:25:13.636175 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:13.636128 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:14.257340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:14.257304 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" path="/var/lib/kubelet/pods/4b869789-ca41-4036-9c44-d1fbe68481df/volumes" Oct 31 03:25:14.257807 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:14.257791 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" path="/var/lib/kubelet/pods/952f64f5-b2ba-46d9-b45a-c5d3ab037827/volumes" Oct 31 03:25:14.322723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:14.322686 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:15.033487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:15.032493 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:15.831225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:15.829733 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:15.938083 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:15.938043 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:25:16.031702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.031655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:16.226519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.226483 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:25:16.226758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.226733 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" containerID="cri-o://a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7" gracePeriod=30 Oct 31 03:25:16.228980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.228950 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:25:16.229337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.229308 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" containerID="cri-o://4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838" gracePeriod=30 Oct 31 03:25:16.318634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.318552 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:16.666761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.666721 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:25:16.691775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:16.691733 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:17.744325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:17.744285 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:18.455685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.455640 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:18.612974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.612937 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:18.761507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.758969 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:25:18.761507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.759288 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" containerID="cri-o://6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74" gracePeriod=30 Oct 31 03:25:18.762386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.762024 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:25:18.762386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.762277 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" containerID="cri-o://b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d" gracePeriod=30 Oct 31 03:25:18.764720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.764692 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:25:18.764908 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.764881 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" containerID="cri-o://639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399" gracePeriod=30 Oct 31 03:25:18.838472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:18.838393 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.341800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.341755 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.473219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.473184 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.613663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.613564 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.701061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.701018 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:19.701268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.701111 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:19.752546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.752503 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.885044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:19.884954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:20.359400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:20.359324 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:21.106799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.106750 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:21.338154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.338121 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:21.514010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.513974 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:25:21.579313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.578968 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:25:21.604340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604295 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604375 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604460 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604507 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604536 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604581 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604659 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604746 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.604810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.604784 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\") pod \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\" (UID: \"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42\") " Oct 31 03:25:21.606510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.605913 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.606510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.606251 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:21.612130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.612093 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.612130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.612109 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.612390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.612335 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.614623 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.614585 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "kube-api-access-2vr5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.614769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.614616 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.615320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.615293 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:21.615472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.615339 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token" pod "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" (UID: "c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.705900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.705799 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.705900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.705860 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.705900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.705897 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.705944 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.705979 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706015 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706056 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706102 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706217 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706135 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\") pod \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\" (UID: \"6c17ca69-f4d4-4fa3-9e4b-6d668306b360\") " Oct 31 03:25:21.706614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706266 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.706614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706340 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.706614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706572 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.706614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706584 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706702 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706805 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vr5z\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-kube-api-access-2vr5z\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706828 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706841 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706854 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706867 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706879 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706891 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706906 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706918 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706930 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706942 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706957 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.707124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.706969 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.708469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.708440 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.708576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.708529 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "kube-api-access-6zwzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:21.708686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.708669 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:21.708751 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.708733 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo" pod "6c17ca69-f4d4-4fa3-9e4b-6d668306b360" (UID: "6c17ca69-f4d4-4fa3-9e4b-6d668306b360"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:21.807894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.807853 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.807894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.807886 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.807894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.807897 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwzl\" (UniqueName: \"kubernetes.io/projected/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-kube-api-access-6zwzl\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.808160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.807907 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:21.808160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:21.807916 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c17ca69-f4d4-4fa3-9e4b-6d668306b360-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:22.080131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.080005 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:22.114127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.114093 4556 generic.go:356] "Generic (PLEG): container finished" podID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerID="4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838" exitCode=0 Oct 31 03:25:22.114673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.114144 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" Oct 31 03:25:22.114673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.114176 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" event={"ID":"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42","Type":"ContainerDied","Data":"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838"} Oct 31 03:25:22.114673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.114235 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5" event={"ID":"c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42","Type":"ContainerDied","Data":"a073d755ad06db6f89029e8a73012f3d90d8279386c1ebfb53c712d540ecd0dd"} Oct 31 03:25:22.114673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.114262 4556 scope.go:122] "RemoveContainer" containerID="4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838" Oct 31 03:25:22.115942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.115920 4556 generic.go:356] "Generic (PLEG): container finished" podID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerID="a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7" exitCode=0 Oct 31 03:25:22.116071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.115976 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" event={"ID":"6c17ca69-f4d4-4fa3-9e4b-6d668306b360","Type":"ContainerDied","Data":"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7"} Oct 31 03:25:22.116071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.115988 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" Oct 31 03:25:22.116071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.116018 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9" event={"ID":"6c17ca69-f4d4-4fa3-9e4b-6d668306b360","Type":"ContainerDied","Data":"2a71699192fae01bf450979a0e0f2b94f3ff8dc8a463253551fa887854600f00"} Oct 31 03:25:22.127565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.127483 4556 scope.go:122] "RemoveContainer" containerID="4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838" Oct 31 03:25:22.127897 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:22.127871 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838\": container with ID starting with 4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838 not found: ID does not exist" containerID="4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838" Oct 31 03:25:22.127972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.127908 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838"} err="failed to get container status \"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838\": rpc error: code = NotFound desc = could not find container \"4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838\": container with ID starting with 4747d031e77bf1a9e9aa358ca46e7091bbb055b430581734d549b0a4afdc8838 not found: ID does not exist" Oct 31 03:25:22.127972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.127933 4556 scope.go:122] "RemoveContainer" containerID="a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7" Oct 31 03:25:22.138706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.138678 4556 scope.go:122] "RemoveContainer" containerID="a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7" Oct 31 03:25:22.139095 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:22.139064 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7\": container with ID starting with a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7 not found: ID does not exist" containerID="a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7" Oct 31 03:25:22.139188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.139107 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7"} err="failed to get container status \"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7\": rpc error: code = NotFound desc = could not find container \"a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7\": container with ID starting with a39904ccf683ad005a48233d7e744a2cac5d6e7e88b3ac02efb33fa637fd42b7 not found: ID does not exist" Oct 31 03:25:22.139317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.139297 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:25:22.142518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.142490 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-chwt5"] Oct 31 03:25:22.152698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.152663 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:25:22.157190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.157156 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-924ee5-openshift-default-77884c8c55-8gnj9"] Oct 31 03:25:22.259355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.259328 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" path="/var/lib/kubelet/pods/6c17ca69-f4d4-4fa3-9e4b-6d668306b360/volumes" Oct 31 03:25:22.259903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:22.259885 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" path="/var/lib/kubelet/pods/c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42/volumes" Oct 31 03:25:23.315323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.315284 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:23.394691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.394658 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:23.394874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.394716 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:23.526047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.526009 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:23.715390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.715348 4556 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:25:23.715606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:23.715410 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:25:24.037078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.037049 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:25:24.105244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.105218 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:25:24.109957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.109927 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:25:24.133996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.133961 4556 generic.go:356] "Generic (PLEG): container finished" podID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerID="639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399" exitCode=0 Oct 31 03:25:24.134198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134159 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" event={"ID":"9cae1473-cb7e-4ed9-ab05-51290cb2e06b","Type":"ContainerDied","Data":"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399"} Oct 31 03:25:24.134279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134214 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" event={"ID":"9cae1473-cb7e-4ed9-ab05-51290cb2e06b","Type":"ContainerDied","Data":"3bce468a7b7c84a1327225bc27d15cac7558fe0b204a79493589d2f5be57ad64"} Oct 31 03:25:24.134279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134241 4556 scope.go:122] "RemoveContainer" containerID="639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399" Oct 31 03:25:24.134382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134326 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.134382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134358 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.134382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134369 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp" Oct 31 03:25:24.134590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134394 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.134590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134452 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.134590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134481 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.134590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134509 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.134590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134562 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134610 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134648 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134708 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134753 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134780 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.134843 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134810 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134853 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134883 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134906 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134930 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.134990 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135019 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135043 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135057 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.135071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135071 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135112 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135149 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135175 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\") pod \"2bd134a4-2bb3-479a-b138-75e074a2a039\" (UID: \"2bd134a4-2bb3-479a-b138-75e074a2a039\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135213 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135236 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\") pod \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\" (UID: \"9cae1473-cb7e-4ed9-ab05-51290cb2e06b\") " Oct 31 03:25:24.135437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135294 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\") pod \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\" (UID: \"a1f3270d-a944-4c64-be89-1bc17fcb7a21\") " Oct 31 03:25:24.135868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.135846 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.136769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.136503 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.136769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.136545 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:24.137628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137046 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.137628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137105 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:24.137628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137393 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.137628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137622 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.137864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137806 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.139456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.137998 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.139456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.138184 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.139456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.138375 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.139456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.138679 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.139760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.139608 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.140218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140190 4556 generic.go:356] "Generic (PLEG): container finished" podID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerID="b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d" exitCode=0 Oct 31 03:25:24.140218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140206 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:24.140503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140301 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" Oct 31 03:25:24.140503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140304 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" event={"ID":"2bd134a4-2bb3-479a-b138-75e074a2a039","Type":"ContainerDied","Data":"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d"} Oct 31 03:25:24.140503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140350 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq" event={"ID":"2bd134a4-2bb3-479a-b138-75e074a2a039","Type":"ContainerDied","Data":"2b2c8ba118aed02d3ca05a08e527542da95ab7eff1352c89d4d3cd80f5d65a0a"} Oct 31 03:25:24.140759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.140502 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.142624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.142596 4556 generic.go:356] "Generic (PLEG): container finished" podID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerID="6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74" exitCode=0 Oct 31 03:25:24.142732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.142661 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerDied","Data":"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74"} Oct 31 03:25:24.142732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.142671 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" Oct 31 03:25:24.142732 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.142702 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg" event={"ID":"a1f3270d-a944-4c64-be89-1bc17fcb7a21","Type":"ContainerDied","Data":"1f0af2ee73ca0d12f3dafd2bd082e7bcb10c9f784d6fbf999663676407eda5c8"} Oct 31 03:25:24.144068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.143954 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.144068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.143971 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:24.144068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144032 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.144307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144072 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "kube-api-access-s8npf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.144307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144142 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:24.144307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144174 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.144307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144296 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "kube-api-access-gwlpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.144554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144294 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token" pod "9cae1473-cb7e-4ed9-ab05-51290cb2e06b" (UID: "9cae1473-cb7e-4ed9-ab05-51290cb2e06b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.144620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.144547 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:25:24.145050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.145017 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:24.145870 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.145851 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token" pod "2bd134a4-2bb3-479a-b138-75e074a2a039" (UID: "2bd134a4-2bb3-479a-b138-75e074a2a039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.146833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.146809 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf" pod "a1f3270d-a944-4c64-be89-1bc17fcb7a21" (UID: "a1f3270d-a944-4c64-be89-1bc17fcb7a21"). InnerVolumeSpecName "kube-api-access-bkcpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:24.158630 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.158600 4556 scope.go:122] "RemoveContainer" containerID="639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399" Oct 31 03:25:24.159108 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:24.159038 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399\": container with ID starting with 639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399 not found: ID does not exist" containerID="639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399" Oct 31 03:25:24.159238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.159119 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399"} err="failed to get container status \"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399\": rpc error: code = NotFound desc = could not find container \"639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399\": container with ID starting with 639536d10a56dfe939d83551357c03d556b9dfecc261dc2c87a654e262e78399 not found: ID does not exist" Oct 31 03:25:24.159238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.159153 4556 scope.go:122] "RemoveContainer" containerID="b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d" Oct 31 03:25:24.175898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.175867 4556 scope.go:122] "RemoveContainer" containerID="b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d" Oct 31 03:25:24.176280 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:24.176262 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d\": container with ID starting with b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d not found: ID does not exist" containerID="b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d" Oct 31 03:25:24.176388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.176290 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d"} err="failed to get container status \"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d\": rpc error: code = NotFound desc = could not find container \"b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d\": container with ID starting with b80f16ca3be3396ecc5eb687b43d84bdfeec3679b038461eca3f5c7200b8102d not found: ID does not exist" Oct 31 03:25:24.176388 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.176310 4556 scope.go:122] "RemoveContainer" containerID="6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74" Oct 31 03:25:24.186523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.186496 4556 scope.go:122] "RemoveContainer" containerID="feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691" Oct 31 03:25:24.196191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.196168 4556 scope.go:122] "RemoveContainer" containerID="6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74" Oct 31 03:25:24.196537 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:24.196507 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74\": container with ID starting with 6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74 not found: ID does not exist" containerID="6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74" Oct 31 03:25:24.196592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.196545 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74"} err="failed to get container status \"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74\": rpc error: code = NotFound desc = could not find container \"6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74\": container with ID starting with 6a5e8e2adb46d4bed8da78d7e8dfd6eb173519a91886ebdcfba57c685621cb74 not found: ID does not exist" Oct 31 03:25:24.196592 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.196566 4556 scope.go:122] "RemoveContainer" containerID="feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691" Oct 31 03:25:24.196864 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:24.196844 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691\": container with ID starting with feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691 not found: ID does not exist" containerID="feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691" Oct 31 03:25:24.196925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.196874 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691"} err="failed to get container status \"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691\": rpc error: code = NotFound desc = could not find container \"feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691\": container with ID starting with feb487f43933424b8960df76980a74702fd39b21463fd51e5f866124e9c60691 not found: ID does not exist" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237048 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237083 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwlpd\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-kube-api-access-gwlpd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237096 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237106 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8npf\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-kube-api-access-s8npf\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237119 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237129 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237137 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237145 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237153 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237161 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237171 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237180 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237189 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237197 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237206 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237214 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237223 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237230 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237239 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bd134a4-2bb3-479a-b138-75e074a2a039-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237247 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237255 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237263 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f3270d-a944-4c64-be89-1bc17fcb7a21-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237271 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkcpf\" (UniqueName: \"kubernetes.io/projected/a1f3270d-a944-4c64-be89-1bc17fcb7a21-kube-api-access-bkcpf\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237279 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bd134a4-2bb3-479a-b138-75e074a2a039-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237286 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.237524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.237294 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cae1473-cb7e-4ed9-ab05-51290cb2e06b-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:24.451798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.451759 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:25:24.455547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.455508 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:24.457396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.457371 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-dvsnp"] Oct 31 03:25:24.467187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.467151 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:25:24.471492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.471460 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-kdzcq"] Oct 31 03:25:24.481194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.481159 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:25:24.487748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:24.487611 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2206e7-openshift-default-69ffc4df47-cgrbg"] Oct 31 03:25:25.141966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:25.141091 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:25.927646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:25.927608 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:26.256833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:26.256754 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" path="/var/lib/kubelet/pods/2bd134a4-2bb3-479a-b138-75e074a2a039/volumes" Oct 31 03:25:26.257241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:26.257227 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" path="/var/lib/kubelet/pods/9cae1473-cb7e-4ed9-ab05-51290cb2e06b/volumes" Oct 31 03:25:26.257687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:26.257673 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" path="/var/lib/kubelet/pods/a1f3270d-a944-4c64-be89-1bc17fcb7a21/volumes" Oct 31 03:25:26.628873 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:26.628789 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:27.319880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:27.319836 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:28.029119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:28.029079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:28.400799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:28.400770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:28.725145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:28.725029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:29.432209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:29.431960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:29.573225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:29.573189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:30.153818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:30.153776 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:30.711574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:30.711455 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:25:30.713076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:30.711852 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" podUID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" containerName="maas-api" containerID="cri-o://845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677" gracePeriod=30 Oct 31 03:25:30.794988 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:30.794955 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:30.960874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:30.960841 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:31.020604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.020574 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:25:31.093080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.093045 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:31.137186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137147 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\") pod \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " Oct 31 03:25:31.137367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137238 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\") pod \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " Oct 31 03:25:31.137367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137318 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\" (UniqueName: \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\") pod \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " Oct 31 03:25:31.137367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137352 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\") pod \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\" (UID: \"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8\") " Oct 31 03:25:31.137811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137783 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca" pod "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" (UID: "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:31.137955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.137930 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:31.139508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.139480 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs" pod "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" (UID: "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:31.139580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.139504 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls" pod "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" (UID: "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:31.139652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.139637 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f" pod "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" (UID: "7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8"). InnerVolumeSpecName "kube-api-access-cjg9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:31.186544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.186506 4556 generic.go:356] "Generic (PLEG): container finished" podID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" containerID="845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677" exitCode=0 Oct 31 03:25:31.186734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.186545 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" event={"ID":"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8","Type":"ContainerDied","Data":"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677"} Oct 31 03:25:31.186734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.186602 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" event={"ID":"7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8","Type":"ContainerDied","Data":"eff0849e660ab9a0361bde5f345e91deffba50db81c63b11e08fed6e136350eb"} Oct 31 03:25:31.186734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.186626 4556 scope.go:122] "RemoveContainer" containerID="845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677" Oct 31 03:25:31.186734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.186553 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp" Oct 31 03:25:31.197825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.197801 4556 scope.go:122] "RemoveContainer" containerID="845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677" Oct 31 03:25:31.198189 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:31.198165 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677\": container with ID starting with 845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677 not found: ID does not exist" containerID="845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677" Oct 31 03:25:31.198310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.198197 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677"} err="failed to get container status \"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677\": rpc error: code = NotFound desc = could not find container \"845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677\": container with ID starting with 845aa202c333241c2d291f820b737ea1327bec47d1797584dbf4010bcb956677 not found: ID does not exist" Oct 31 03:25:31.208212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.208177 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:25:31.212331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.212302 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-0a531d-869c595655-gdmjp"] Oct 31 03:25:31.239499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.239457 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:31.239499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.239494 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:31.239499 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.239507 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjg9f\" (UniqueName: \"kubernetes.io/projected/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8-kube-api-access-cjg9f\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:31.545845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.545808 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:31.927123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:31.927085 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:32.258347 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:32.258266 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" path="/var/lib/kubelet/pods/7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8/volumes" Oct 31 03:25:32.632741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:32.632643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:33.259278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:33.259231 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:25:33.415586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:33.415544 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:34.063993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:34.063954 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:34.826309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:34.826278 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:35.540773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:35.540644 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:36.335742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:36.335701 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:37.046047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:37.046002 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:37.737281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:37.737237 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:38.429238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:38.429194 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:38.596358 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:38.596328 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:38.726478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:38.726380 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:39.024174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.024077 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:39.252262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.252223 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:39.485597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.485567 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:39.595556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.595518 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz"] Oct 31 03:25:39.595969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.595953 4556 cpu_manager.go:414] "containerMap: removing container" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.595970 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.595988 4556 cpu_manager.go:414] "containerMap: removing container" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.595993 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596001 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596005 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" Oct 31 03:25:39.596014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596014 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596018 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596025 4556 cpu_manager.go:414] "containerMap: removing container" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596030 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596042 4556 cpu_manager.go:414] "containerMap: removing container" podUID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" containerName="payload-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596047 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" containerName="payload-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596054 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596058 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596064 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596068 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596077 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596081 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596090 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596094 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596101 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596106 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596114 4556 cpu_manager.go:414] "containerMap: removing container" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596119 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596126 4556 cpu_manager.go:414] "containerMap: removing container" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596131 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596136 4556 cpu_manager.go:414] "containerMap: removing container" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596141 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596147 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596151 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596158 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596162 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596171 4556 cpu_manager.go:414] "containerMap: removing container" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596175 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596182 4556 cpu_manager.go:414] "containerMap: removing container" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596186 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596193 4556 cpu_manager.go:414] "containerMap: removing container" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596197 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596203 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" Oct 31 03:25:39.596193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596208 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596216 4556 cpu_manager.go:414] "containerMap: removing container" podUID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596220 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596227 4556 cpu_manager.go:414] "containerMap: removing container" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596231 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07829292-1d8e-4032-91e6-6fa8d3fffcb3" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596238 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596242 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596249 4556 cpu_manager.go:414] "containerMap: removing container" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596253 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596259 4556 cpu_manager.go:414] "containerMap: removing container" podUID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596263 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596269 4556 cpu_manager.go:414] "containerMap: removing container" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596273 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596280 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9840d12a-89c6-44d4-976d-5b32edd8a23d" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596284 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9840d12a-89c6-44d4-976d-5b32edd8a23d" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596291 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596295 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596305 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596309 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596364 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d225d9d-9b90-49ba-9ad1-ffbf1fd56a4f" containerName="payload-pre-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596373 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596379 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3ab0f9-98e5-42b6-95fb-86317586e85c" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596386 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596392 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="70af976b-5785-4f55-ba46-c652c94a56c0" containerName="main" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596397 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65a5e49-14c2-4907-b295-6ccc473195b5" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596402 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd134a4-2bb3-479a-b138-75e074a2a039" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596408 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f16360-e558-4069-99ff-a4cfd951257f" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596442 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="405f6ada-3a66-45f1-acd8-64f2729f4ba2" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596448 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596453 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba3de21-6f99-4686-838c-4b35ea81602c" containerName="payload-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596459 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf8e5ea-0abb-4d89-9a75-7f6def8bc8a8" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596465 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cae1473-cb7e-4ed9-ab05-51290cb2e06b" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596471 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee7dd0d-24dc-457c-ad21-38a84b07f1b9" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596477 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f103c4-8022-4f07-a6be-b76dfa9f1ccc" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596482 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="137b2f24-7930-414e-acec-df30d6a1f713" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596487 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9840d12a-89c6-44d4-976d-5b32edd8a23d" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596491 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4baf11b-8eff-4af3-b6f5-3f9c4f46bd42" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596497 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa98cdf-5d23-4e37-8c63-6e8619231161" containerName="maas-api" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596503 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="952f64f5-b2ba-46d9-b45a-c5d3ab037827" containerName="payload-pre-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596509 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedc3cc2-fb07-49bb-a553-3283341edeaf" containerName="payload-pre-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596515 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b702eeb-fa9e-460b-8c81-1feeea666d69" containerName="main" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596520 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a20ec2-690a-4360-9287-a2f8f85f09b4" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596525 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefa6a8e-9cfa-4809-83be-a2ba264b914c" containerName="payload-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596530 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b869789-ca41-4036-9c44-d1fbe68481df" containerName="payload-processing" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596536 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c17ca69-f4d4-4fa3-9e4b-6d668306b360" containerName="istio-proxy" Oct 31 03:25:39.597189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.596541 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e66071d-8b8e-4713-b328-5b86e2510379" containerName="istio-proxy" Oct 31 03:25:39.600880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.600860 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.606314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.606282 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:25:39.606591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.606335 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:25:39.609034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.608834 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz"] Oct 31 03:25:39.610069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.610042 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:25:39.610214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.610115 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:25:39.726925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.726883 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9llw\" (UniqueName: \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.727143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.726983 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.727199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.727132 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.827863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.827764 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.827863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.827836 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9llw\" (UniqueName: \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.828076 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.827888 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.828517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.828497 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.830261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.830233 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.839359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.839322 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9llw\" (UniqueName: \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\") pod \"maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:39.917249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:39.917200 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:40.195236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:40.195196 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz"] Oct 31 03:25:40.198510 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:25:40.198478 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e31042_3372_4b86_ac39_8b41352d3ddc.slice/crio-ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579 WatchSource:0}: Error finding container ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579: Status 404 returned error can't find the container with id ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579 Oct 31 03:25:40.209950 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:40.209916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:40.237792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:40.237742 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" event={"ID":"11e31042-3372-4b86-ac39-8b41352d3ddc","Type":"ContainerStarted","Data":"ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579"} Oct 31 03:25:40.943629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:40.943594 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:41.243847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:41.243750 4556 generic.go:356] "Generic (PLEG): container finished" podID="11e31042-3372-4b86-ac39-8b41352d3ddc" containerID="61f89b928cfc78421cd979bc55c0060e03151cb6f8e9abf46d384d8b5fc712ad" exitCode=0 Oct 31 03:25:41.244230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:41.243844 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" event={"ID":"11e31042-3372-4b86-ac39-8b41352d3ddc","Type":"ContainerDied","Data":"61f89b928cfc78421cd979bc55c0060e03151cb6f8e9abf46d384d8b5fc712ad"} Oct 31 03:25:41.718589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:41.718554 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:42.404813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.404781 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:42.421120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.420388 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:42.556474 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.556367 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\") pod \"11e31042-3372-4b86-ac39-8b41352d3ddc\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " Oct 31 03:25:42.556657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.556495 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\" (UniqueName: \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\") pod \"11e31042-3372-4b86-ac39-8b41352d3ddc\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " Oct 31 03:25:42.556657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.556578 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\") pod \"11e31042-3372-4b86-ac39-8b41352d3ddc\" (UID: \"11e31042-3372-4b86-ac39-8b41352d3ddc\") " Oct 31 03:25:42.556930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.556904 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca" pod "11e31042-3372-4b86-ac39-8b41352d3ddc" (UID: "11e31042-3372-4b86-ac39-8b41352d3ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:42.558658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.558629 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca" pod "11e31042-3372-4b86-ac39-8b41352d3ddc" (UID: "11e31042-3372-4b86-ac39-8b41352d3ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:42.558773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.558728 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw" pod "11e31042-3372-4b86-ac39-8b41352d3ddc" (UID: "11e31042-3372-4b86-ac39-8b41352d3ddc"). InnerVolumeSpecName "kube-api-access-z9llw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:42.657739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.657702 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9llw\" (UniqueName: \"kubernetes.io/projected/11e31042-3372-4b86-ac39-8b41352d3ddc-kube-api-access-z9llw\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:42.657739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.657735 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:42.657739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:42.657748 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e31042-3372-4b86-ac39-8b41352d3ddc-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:43.181349 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:43.181307 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:43.255052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:43.255022 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" Oct 31 03:25:43.255052 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:43.255035 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz" event={"ID":"11e31042-3372-4b86-ac39-8b41352d3ddc","Type":"ContainerDied","Data":"ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579"} Oct 31 03:25:43.255391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:43.255077 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea354d25d5742f877756cfd6f8d6b448a44cdcae81faa600e1515b334fe09579" Oct 31 03:25:43.843811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:43.843770 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:44.398377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:44.398335 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:25:44.405457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:44.405405 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:25:44.410237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:44.410195 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:25:44.538004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:44.537961 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:44.673091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:44.673048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:45.231029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:45.230988 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:45.939101 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:25:45.939059 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:25:45.944221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:45.944184 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:46.743710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:46.743665 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:47.429774 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:47.429305 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.133008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:48.132970 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.915742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:48.915691 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:49.622428 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:49.622205 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:49.715617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:49.715538 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:25:49.728674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:49.728635 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:25:49.823304 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:49.823270 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:50.026761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.026723 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:50.338199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.338108 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:50.831283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.831242 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz"] Oct 31 03:25:50.831959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.831936 4556 cpu_manager.go:414] "containerMap: removing container" podUID="11e31042-3372-4b86-ac39-8b41352d3ddc" containerName="revoke-keys" Oct 31 03:25:50.832067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.831961 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e31042-3372-4b86-ac39-8b41352d3ddc" containerName="revoke-keys" Oct 31 03:25:50.832128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.832093 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f3270d-a944-4c64-be89-1bc17fcb7a21" containerName="istio-proxy" Oct 31 03:25:50.832128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.832109 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e31042-3372-4b86-ac39-8b41352d3ddc" containerName="revoke-keys" Oct 31 03:25:50.835667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.835641 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.838717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.838690 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:25:50.838942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.838918 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:25:50.839071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.838981 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:25:50.839128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.839075 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:25:50.844649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.844624 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz"] Oct 31 03:25:50.851632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.851597 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.851824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.851755 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.851877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.851834 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlth\" (UniqueName: \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.953231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.953182 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.953496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.953258 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.953496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.953287 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlth\" (UniqueName: \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.954062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.954026 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.955709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.955684 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:50.965290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:50.965262 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlth\" (UniqueName: \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\") pod \"maas-api-revoke-sub-1fa83ed4-fsztz\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:51.152273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:51.152238 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:51.431509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:51.431479 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz"] Oct 31 03:25:51.433811 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:25:51.433782 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6fd3aa_37ee_4806_96b3_e46df3a96b22.slice/crio-31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842 WatchSource:0}: Error finding container 31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842: Status 404 returned error can't find the container with id 31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842 Oct 31 03:25:52.246061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:52.246020 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.303155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:52.303118 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" event={"ID":"2c6fd3aa-37ee-4806-96b3-e46df3a96b22","Type":"ContainerStarted","Data":"26745c117af09a92812f1d08c4cb59d59252fc9039b81495df002b166159fadf"} Oct 31 03:25:52.303361 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:52.303175 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" event={"ID":"2c6fd3aa-37ee-4806-96b3-e46df3a96b22","Type":"ContainerStarted","Data":"31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842"} Oct 31 03:25:52.322572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:52.322512 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" podStartSLOduration=2.322497039 podStartE2EDuration="2.322497039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:52.320882835 +0000 UTC m=+1830.175899976" watchObservedRunningTime="2026-10-31 03:25:52.322497039 +0000 UTC m=+1830.177514228" Oct 31 03:25:52.382230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:52.382197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.309221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:53.309183 4556 generic.go:356] "Generic (PLEG): container finished" podID="2c6fd3aa-37ee-4806-96b3-e46df3a96b22" containerID="26745c117af09a92812f1d08c4cb59d59252fc9039b81495df002b166159fadf" exitCode=0 Oct 31 03:25:53.309664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:53.309272 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" event={"ID":"2c6fd3aa-37ee-4806-96b3-e46df3a96b22","Type":"ContainerDied","Data":"26745c117af09a92812f1d08c4cb59d59252fc9039b81495df002b166159fadf"} Oct 31 03:25:54.471264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.471233 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:54.493265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.493231 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\") pod \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " Oct 31 03:25:54.493467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.493332 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\" (UniqueName: \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\") pod \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " Oct 31 03:25:54.493467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.493354 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\") pod \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\" (UID: \"2c6fd3aa-37ee-4806-96b3-e46df3a96b22\") " Oct 31 03:25:54.493776 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.493751 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca" pod "2c6fd3aa-37ee-4806-96b3-e46df3a96b22" (UID: "2c6fd3aa-37ee-4806-96b3-e46df3a96b22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:25:54.495567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.495536 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca" pod "2c6fd3aa-37ee-4806-96b3-e46df3a96b22" (UID: "2c6fd3aa-37ee-4806-96b3-e46df3a96b22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:54.496032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.496005 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth" pod "2c6fd3aa-37ee-4806-96b3-e46df3a96b22" (UID: "2c6fd3aa-37ee-4806-96b3-e46df3a96b22"). InnerVolumeSpecName "kube-api-access-pdlth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:54.595017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.594926 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:54.595017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.594962 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlth\" (UniqueName: \"kubernetes.io/projected/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-kube-api-access-pdlth\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:54.595017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.594974 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6fd3aa-37ee-4806-96b3-e46df3a96b22-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:54.716293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.716241 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" containerID="cri-o://91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c" gracePeriod=25 Oct 31 03:25:54.729583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:54.729530 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" containerID="cri-o://167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f" gracePeriod=25 Oct 31 03:25:55.320319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:55.320284 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" Oct 31 03:25:55.320544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:55.320314 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz" event={"ID":"2c6fd3aa-37ee-4806-96b3-e46df3a96b22","Type":"ContainerDied","Data":"31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842"} Oct 31 03:25:55.320544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:55.320380 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31092f619ae623d68c0dd673ddd14a3a54cab8c600eda05fe5165723d7b42842" Oct 31 03:25:59.421610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.421568 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" containerID="cri-o://503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112" gracePeriod=45 Oct 31 03:25:59.438409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.438288 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" containerID="cri-o://9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6" gracePeriod=45 Oct 31 03:25:59.438409 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.438303 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" containerID="cri-o://9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5" gracePeriod=45 Oct 31 03:25:59.704930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.704886 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:25:59.746014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.745984 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:25:59.749482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.749453 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750328 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750385 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750464 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750563 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\" (UniqueName: \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750595 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\") pod \"2269b73b-86ae-4697-be99-cffa142e11ef\" (UID: \"2269b73b-86ae-4697-be99-cffa142e11ef\") " Oct 31 03:25:59.750685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750627 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.751004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.750833 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.751068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.751055 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.751119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.751077 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.751335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.751310 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.752765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.752738 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.753852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.753818 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "kube-api-access-bz4b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:59.754189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.754108 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:25:59.755090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.755068 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs" pod "2269b73b-86ae-4697-be99-cffa142e11ef" (UID: "2269b73b-86ae-4697-be99-cffa142e11ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:59.852024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.851984 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852053 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852073 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852102 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\" (UniqueName: \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852152 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852200 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852278 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852306 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\" (UniqueName: \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852333 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852375 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852430 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\") pod \"c7d01c7f-861a-47d6-a69d-935c94c338b3\" (UID: \"c7d01c7f-861a-47d6-a69d-935c94c338b3\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852442 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852513 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\") pod \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\" (UID: \"80327f0f-af3d-4a66-b1c1-a05cb396b8fd\") " Oct 31 03:25:59.852648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852579 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852922 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4b6\" (UniqueName: \"kubernetes.io/projected/2269b73b-86ae-4697-be99-cffa142e11ef-kube-api-access-bz4b6\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852946 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852962 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2269b73b-86ae-4697-be99-cffa142e11ef-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852977 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.852990 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2269b73b-86ae-4697-be99-cffa142e11ef-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.853003 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.853145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.853111 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.853568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.853225 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.853568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.853242 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.853568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.853539 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.854943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.854911 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.855103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.855077 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:59.855230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.855210 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:25:59.855230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.855219 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j" pod "80327f0f-af3d-4a66-b1c1-a05cb396b8fd" (UID: "80327f0f-af3d-4a66-b1c1-a05cb396b8fd"). InnerVolumeSpecName "kube-api-access-fj96j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:59.855596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.855563 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "kube-api-access-ddlrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:59.855723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.855621 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs" pod "c7d01c7f-861a-47d6-a69d-935c94c338b3" (UID: "c7d01c7f-861a-47d6-a69d-935c94c338b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953768 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d01c7f-861a-47d6-a69d-935c94c338b3-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953799 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj96j\" (UniqueName: \"kubernetes.io/projected/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-kube-api-access-fj96j\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953811 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953820 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953829 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953838 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953847 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.953853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953856 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80327f0f-af3d-4a66-b1c1-a05cb396b8fd-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.954236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953866 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlrg\" (UniqueName: \"kubernetes.io/projected/c7d01c7f-861a-47d6-a69d-935c94c338b3-kube-api-access-ddlrg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:59.954236 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:25:59.953880 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d01c7f-861a-47d6-a69d-935c94c338b3-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:00.346283 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.346249 4556 generic.go:356] "Generic (PLEG): container finished" podID="2269b73b-86ae-4697-be99-cffa142e11ef" containerID="503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112" exitCode=0 Oct 31 03:26:00.346482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.346297 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" Oct 31 03:26:00.346482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.346331 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" event={"ID":"2269b73b-86ae-4697-be99-cffa142e11ef","Type":"ContainerDied","Data":"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112"} Oct 31 03:26:00.346482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.346392 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r" event={"ID":"2269b73b-86ae-4697-be99-cffa142e11ef","Type":"ContainerDied","Data":"7538f35f6027def0cc947de02ef1584e8823ab3fc839c8fd8540d60dde481da0"} Oct 31 03:26:00.346482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.346435 4556 scope.go:122] "RemoveContainer" containerID="503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112" Oct 31 03:26:00.347717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.347697 4556 generic.go:356] "Generic (PLEG): container finished" podID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerID="9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6" exitCode=0 Oct 31 03:26:00.347815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.347749 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" Oct 31 03:26:00.347815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.347781 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" event={"ID":"c7d01c7f-861a-47d6-a69d-935c94c338b3","Type":"ContainerDied","Data":"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6"} Oct 31 03:26:00.347933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.347816 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl" event={"ID":"c7d01c7f-861a-47d6-a69d-935c94c338b3","Type":"ContainerDied","Data":"7948398b8fe6a0f9f6424b2ea59ffa78e5a9b1a408abf21bffc7f5f8ad142cc7"} Oct 31 03:26:00.349289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.349264 4556 generic.go:356] "Generic (PLEG): container finished" podID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerID="9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5" exitCode=0 Oct 31 03:26:00.349395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.349293 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" Oct 31 03:26:00.349491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.349402 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" event={"ID":"80327f0f-af3d-4a66-b1c1-a05cb396b8fd","Type":"ContainerDied","Data":"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5"} Oct 31 03:26:00.349491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.349460 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd" event={"ID":"80327f0f-af3d-4a66-b1c1-a05cb396b8fd","Type":"ContainerDied","Data":"12a7f062dbc1583fcdfeb164cfe92231fd87d28c6dc1505cd5b8fac008436fbe"} Oct 31 03:26:00.357237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.357209 4556 scope.go:122] "RemoveContainer" containerID="503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112" Oct 31 03:26:00.357595 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:00.357573 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112\": container with ID starting with 503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112 not found: ID does not exist" containerID="503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112" Oct 31 03:26:00.357659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.357605 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112"} err="failed to get container status \"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112\": rpc error: code = NotFound desc = could not find container \"503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112\": container with ID starting with 503d9f08469dc7b3ae6cf8532f748df001ead930279e019c48008a6b36416112 not found: ID does not exist" Oct 31 03:26:00.357659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.357623 4556 scope.go:122] "RemoveContainer" containerID="9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6" Oct 31 03:26:00.366699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.366667 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:26:00.367665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.367643 4556 scope.go:122] "RemoveContainer" containerID="9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6" Oct 31 03:26:00.368124 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:00.368097 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6\": container with ID starting with 9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6 not found: ID does not exist" containerID="9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6" Oct 31 03:26:00.368221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.368131 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6"} err="failed to get container status \"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6\": rpc error: code = NotFound desc = could not find container \"9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6\": container with ID starting with 9802b87166ae6f5df39d1ed2eb23bb92dc97f1f2292643a9dee98a1bea4b90d6 not found: ID does not exist" Oct 31 03:26:00.368221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.368150 4556 scope.go:122] "RemoveContainer" containerID="9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5" Oct 31 03:26:00.370325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.370302 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-2-w4-bebea4-kserve-c95455c75-jrn5r"] Oct 31 03:26:00.377648 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.377627 4556 scope.go:122] "RemoveContainer" containerID="9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5" Oct 31 03:26:00.378039 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:00.378004 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5\": container with ID starting with 9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5 not found: ID does not exist" containerID="9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5" Oct 31 03:26:00.378204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.378049 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5"} err="failed to get container status \"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5\": rpc error: code = NotFound desc = could not find container \"9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5\": container with ID starting with 9f03464a9ac1a02bc9b527d5d0aa7a86f6228cfef6c51740a034cd93d7e6a2e5 not found: ID does not exist" Oct 31 03:26:00.379590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.379566 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:26:00.383367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.383338 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/premium-w4-bebea4-kserve-75f9cbb589-72qzd"] Oct 31 03:26:00.392364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.392329 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:26:00.396704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.396675 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebea4/distinct-w4-bebea4-kserve-676c44d988-h8pfl"] Oct 31 03:26:00.893617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:00.893576 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:26:01.901072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:01.901028 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:26:02.256508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:02.256403 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" path="/var/lib/kubelet/pods/2269b73b-86ae-4697-be99-cffa142e11ef/volumes" Oct 31 03:26:02.256877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:02.256862 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" path="/var/lib/kubelet/pods/80327f0f-af3d-4a66-b1c1-a05cb396b8fd/volumes" Oct 31 03:26:02.257232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:02.257221 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" path="/var/lib/kubelet/pods/c7d01c7f-861a-47d6-a69d-935c94c338b3/volumes" Oct 31 03:26:03.107431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:03.107368 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:26:05.971258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:05.971152 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.091810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.091770 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz"] Oct 31 03:26:06.096466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.096433 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-bebea4-wz7lz"] Oct 31 03:26:06.118719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.118688 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.256985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.256903 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e31042-3372-4b86-ac39-8b41352d3ddc" path="/var/lib/kubelet/pods/11e31042-3372-4b86-ac39-8b41352d3ddc/volumes" Oct 31 03:26:06.258162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.258136 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.685522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:06.685489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:08.828260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:08.828217 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:26:09.585287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:09.585249 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:10.248671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:10.248638 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:10.417700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:10.417666 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:10.892990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:10.892952 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:26:11.901573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:11.901532 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:26:12.798961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:12.798926 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:13.007656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:13.007612 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:13.101507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:13.101368 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:26:13.101937 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:13.101910 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" containerID="cri-o://77aabc95556893e46b9651acf7cea857ea9cc2e283d7c1f0a10e2d3f5188ee39" gracePeriod=30 Oct 31 03:26:13.155192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:13.155151 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:14.980341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:14.980268 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:26:15.010327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.010298 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:26:15.027104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.027064 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\") pod \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " Oct 31 03:26:15.027104 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.027111 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\") pod \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " Oct 31 03:26:15.030401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.027194 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\" (UniqueName: \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\") pod \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\" (UID: \"b26adffd-9b62-4b0d-9f8f-32d994efb4d4\") " Oct 31 03:26:15.030401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.027257 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\" (UniqueName: \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\") pod \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\" (UID: \"d5ab1005-92d1-4af6-ae20-b691ce4378a8\") " Oct 31 03:26:15.030401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.027963 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume" pod "d5ab1005-92d1-4af6-ae20-b691ce4378a8" (UID: "d5ab1005-92d1-4af6-ae20-b691ce4378a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:15.030401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.028492 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ab1005-92d1-4af6-ae20-b691ce4378a8-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:15.030769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.030673 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n" pod "d5ab1005-92d1-4af6-ae20-b691ce4378a8" (UID: "d5ab1005-92d1-4af6-ae20-b691ce4378a8"). InnerVolumeSpecName "kube-api-access-62h6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:15.030769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.030699 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume" pod "b26adffd-9b62-4b0d-9f8f-32d994efb4d4" (UID: "b26adffd-9b62-4b0d-9f8f-32d994efb4d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:15.031158 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.031127 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz" pod "b26adffd-9b62-4b0d-9f8f-32d994efb4d4" (UID: "b26adffd-9b62-4b0d-9f8f-32d994efb4d4"). InnerVolumeSpecName "kube-api-access-rqfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:15.129893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.129850 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:15.129893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.129888 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfwz\" (UniqueName: \"kubernetes.io/projected/b26adffd-9b62-4b0d-9f8f-32d994efb4d4-kube-api-access-rqfwz\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:15.129893 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.129898 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62h6n\" (UniqueName: \"kubernetes.io/projected/d5ab1005-92d1-4af6-ae20-b691ce4378a8-kube-api-access-62h6n\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:15.429904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.429861 4556 generic.go:356] "Generic (PLEG): container finished" podID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerID="167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f" exitCode=0 Oct 31 03:26:15.430122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.429929 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" Oct 31 03:26:15.430122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.429986 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" event={"ID":"b26adffd-9b62-4b0d-9f8f-32d994efb4d4","Type":"ContainerDied","Data":"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f"} Oct 31 03:26:15.430122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.430083 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg" event={"ID":"b26adffd-9b62-4b0d-9f8f-32d994efb4d4","Type":"ContainerDied","Data":"7d73aba75f7131f88b806eede4c3714eb69fbe6598ade1a1e4d0caf3ff720502"} Oct 31 03:26:15.430122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.430113 4556 scope.go:122] "RemoveContainer" containerID="167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f" Oct 31 03:26:15.431599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.431571 4556 generic.go:356] "Generic (PLEG): container finished" podID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerID="91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c" exitCode=0 Oct 31 03:26:15.431756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.431624 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" event={"ID":"d5ab1005-92d1-4af6-ae20-b691ce4378a8","Type":"ContainerDied","Data":"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c"} Oct 31 03:26:15.431756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.431644 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" Oct 31 03:26:15.431756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.431677 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq" event={"ID":"d5ab1005-92d1-4af6-ae20-b691ce4378a8","Type":"ContainerDied","Data":"771018affd0f6d354feca560d14beafc27fc19c48c36ca03345417e68fc85565"} Oct 31 03:26:15.440997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.440965 4556 scope.go:122] "RemoveContainer" containerID="167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f" Oct 31 03:26:15.441472 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:15.441439 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f\": container with ID starting with 167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f not found: ID does not exist" containerID="167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f" Oct 31 03:26:15.441589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.441485 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f"} err="failed to get container status \"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f\": rpc error: code = NotFound desc = could not find container \"167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f\": container with ID starting with 167ea22509662c21cdc06bfc318856c43ab089e290d314007ea97a4f643a3d9f not found: ID does not exist" Oct 31 03:26:15.441589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.441512 4556 scope.go:122] "RemoveContainer" containerID="91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c" Oct 31 03:26:15.450703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.450683 4556 scope.go:122] "RemoveContainer" containerID="91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c" Oct 31 03:26:15.451031 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:15.451011 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c\": container with ID starting with 91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c not found: ID does not exist" containerID="91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c" Oct 31 03:26:15.451074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.451042 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c"} err="failed to get container status \"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c\": rpc error: code = NotFound desc = could not find container \"91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c\": container with ID starting with 91f185dc2303e09ca0cb4d11489fc63ba5b3f1354b98a5209d774669fc05b62c not found: ID does not exist" Oct 31 03:26:15.456933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.456906 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:26:15.462961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.462932 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-bebea4-75f8674b47-9p8dg"] Oct 31 03:26:15.472109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.472074 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:26:15.479125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.479093 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-bebea4-698449c44f-v74mq"] Oct 31 03:26:15.592521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.592482 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:15.738467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:15.738430 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:15.936858 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:15.936820 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:26:16.256426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:16.256376 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" path="/var/lib/kubelet/pods/b26adffd-9b62-4b0d-9f8f-32d994efb4d4/volumes" Oct 31 03:26:16.256818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:16.256801 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" path="/var/lib/kubelet/pods/d5ab1005-92d1-4af6-ae20-b691ce4378a8/volumes" Oct 31 03:26:18.499510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.491405 4556 generic.go:356] "Generic (PLEG): container finished" podID="732eaee1-4e6b-4467-a652-659ef878f820" containerID="77aabc95556893e46b9651acf7cea857ea9cc2e283d7c1f0a10e2d3f5188ee39" exitCode=0 Oct 31 03:26:18.519518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.501534 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" event={"ID":"732eaee1-4e6b-4467-a652-659ef878f820","Type":"ContainerDied","Data":"77aabc95556893e46b9651acf7cea857ea9cc2e283d7c1f0a10e2d3f5188ee39"} Oct 31 03:26:18.666517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.666449 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.789736 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.789840 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.789876 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.789935 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.789977 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.790069 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.790109 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.790148 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.791445 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.790248 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\") pod \"732eaee1-4e6b-4467-a652-659ef878f820\" (UID: \"732eaee1-4e6b-4467-a652-659ef878f820\") " Oct 31 03:26:18.792153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.791817 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.806457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.803538 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.830621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.811779 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.830621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.811860 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.830621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.815734 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:18.844518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.844088 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:26:18.847089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.847028 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.847401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.847374 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:26:18.850440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.848054 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5" pod "732eaee1-4e6b-4467-a652-659ef878f820" (UID: "732eaee1-4e6b-4467-a652-659ef878f820"). InnerVolumeSpecName "kube-api-access-9drl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894128 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732eaee1-4e6b-4467-a652-659ef878f820-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894178 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894195 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894210 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894225 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894242 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894256 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9drl5\" (UniqueName: \"kubernetes.io/projected/732eaee1-4e6b-4467-a652-659ef878f820-kube-api-access-9drl5\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894272 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732eaee1-4e6b-4467-a652-659ef878f820-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:18.894328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:18.894286 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732eaee1-4e6b-4467-a652-659ef878f820-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:19.532243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:19.532141 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" event={"ID":"732eaee1-4e6b-4467-a652-659ef878f820","Type":"ContainerDied","Data":"2e3062a3c97f6592f56463def8ec9f3f1ddd32fd966712bbebd52ce58df02e10"} Oct 31 03:26:19.532877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:19.532854 4556 scope.go:122] "RemoveContainer" containerID="77aabc95556893e46b9651acf7cea857ea9cc2e283d7c1f0a10e2d3f5188ee39" Oct 31 03:26:19.533129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:19.533110 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn" Oct 31 03:26:19.562541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:19.562506 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:26:19.572449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:19.569118 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebea4-openshift-default-7c9c5d994c-jdtgn"] Oct 31 03:26:20.116768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:20.116722 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:20.251468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:20.251385 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:20.256396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:20.256363 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732eaee1-4e6b-4467-a652-659ef878f820" path="/var/lib/kubelet/pods/732eaee1-4e6b-4467-a652-659ef878f820/volumes" Oct 31 03:26:21.496085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:21.496040 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:21.625879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:21.625844 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:21.763938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:21.763798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:22.577367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:22.577329 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:28.745836 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:28.745796 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:34.406837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:34.406799 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:34.544471 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:34.544402 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:39.636624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.636588 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf"] Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.636990 4556 cpu_manager.go:414] "containerMap: removing container" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637000 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637015 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637021 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637032 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637037 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" Oct 31 03:26:39.637046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637047 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637053 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637062 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637066 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637076 4556 cpu_manager.go:414] "containerMap: removing container" podUID="2c6fd3aa-37ee-4806-96b3-e46df3a96b22" containerName="revoke-keys" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637080 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6fd3aa-37ee-4806-96b3-e46df3a96b22" containerName="revoke-keys" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637093 4556 cpu_manager.go:414] "containerMap: removing container" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637097 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637153 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="732eaee1-4e6b-4467-a652-659ef878f820" containerName="istio-proxy" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637162 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26adffd-9b62-4b0d-9f8f-32d994efb4d4" containerName="payload-pre-processing" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637168 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d01c7f-861a-47d6-a69d-935c94c338b3" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637175 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2269b73b-86ae-4697-be99-cffa142e11ef" containerName="main" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637182 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ab1005-92d1-4af6-ae20-b691ce4378a8" containerName="payload-processing" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637188 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6fd3aa-37ee-4806-96b3-e46df3a96b22" containerName="revoke-keys" Oct 31 03:26:39.637281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.637195 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="80327f0f-af3d-4a66-b1c1-a05cb396b8fd" containerName="main" Oct 31 03:26:39.641722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.641702 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.644594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.644567 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:26:39.644734 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.644588 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:26:39.644783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.644744 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:26:39.644783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.644749 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:26:39.650427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.650379 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf"] Oct 31 03:26:39.741153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.741115 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.741328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.741235 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xjk\" (UniqueName: \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.741328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.741281 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.841888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.841842 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xjk\" (UniqueName: \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.842103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.841898 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.842103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.841971 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.842803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.842779 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.844517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.844494 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.854111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.854082 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xjk\" (UniqueName: \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\") pod \"maas-api-revoke-sub-85237e45-4v5kf\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:39.956191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:39.956149 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:40.234087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:40.234055 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf"] Oct 31 03:26:40.236946 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:26:40.236917 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8869ad49_2264_4ddf_895b_9ca8e5a01b3c.slice/crio-ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d WatchSource:0}: Error finding container ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d: Status 404 returned error can't find the container with id ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d Oct 31 03:26:40.633164 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:40.633124 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" event={"ID":"8869ad49-2264-4ddf-895b-9ca8e5a01b3c","Type":"ContainerStarted","Data":"c7340a4100204ddbe37ea955a43f85fc87fe8aed9c5de3fce129c00eafac13a3"} Oct 31 03:26:40.633390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:40.633190 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" event={"ID":"8869ad49-2264-4ddf-895b-9ca8e5a01b3c","Type":"ContainerStarted","Data":"ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d"} Oct 31 03:26:40.651094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:40.651037 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" podStartSLOduration=1.651019856 podStartE2EDuration="1.651019856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:40.649293684 +0000 UTC m=+1878.504310811" watchObservedRunningTime="2026-10-31 03:26:40.651019856 +0000 UTC m=+1878.506036996" Oct 31 03:26:41.639742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:41.639645 4556 generic.go:356] "Generic (PLEG): container finished" podID="8869ad49-2264-4ddf-895b-9ca8e5a01b3c" containerID="c7340a4100204ddbe37ea955a43f85fc87fe8aed9c5de3fce129c00eafac13a3" exitCode=0 Oct 31 03:26:41.640049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:41.639730 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" event={"ID":"8869ad49-2264-4ddf-895b-9ca8e5a01b3c","Type":"ContainerDied","Data":"c7340a4100204ddbe37ea955a43f85fc87fe8aed9c5de3fce129c00eafac13a3"} Oct 31 03:26:42.782631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.782602 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:42.871973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.871933 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\") pod \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " Oct 31 03:26:42.871973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.871982 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\" (UniqueName: \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\") pod \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " Oct 31 03:26:42.872192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.872023 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\") pod \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\" (UID: \"8869ad49-2264-4ddf-895b-9ca8e5a01b3c\") " Oct 31 03:26:42.872508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.872474 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca" pod "8869ad49-2264-4ddf-895b-9ca8e5a01b3c" (UID: "8869ad49-2264-4ddf-895b-9ca8e5a01b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:26:42.874148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.874127 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca" pod "8869ad49-2264-4ddf-895b-9ca8e5a01b3c" (UID: "8869ad49-2264-4ddf-895b-9ca8e5a01b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:26:42.874220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.874154 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk" pod "8869ad49-2264-4ddf-895b-9ca8e5a01b3c" (UID: "8869ad49-2264-4ddf-895b-9ca8e5a01b3c"). InnerVolumeSpecName "kube-api-access-p4xjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:26:42.973308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.973256 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:42.973308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.973297 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xjk\" (UniqueName: \"kubernetes.io/projected/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-kube-api-access-p4xjk\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:42.973308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:42.973307 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8869ad49-2264-4ddf-895b-9ca8e5a01b3c-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:26:43.650523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:43.650483 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" Oct 31 03:26:43.650713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:43.650492 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf" event={"ID":"8869ad49-2264-4ddf-895b-9ca8e5a01b3c","Type":"ContainerDied","Data":"ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d"} Oct 31 03:26:43.650713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:43.650608 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9b4f9f8111c571436d50a5acf5846bbad81c02fff1bc4c3850b9744311a59d" Oct 31 03:26:45.936674 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:26:45.936631 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:26:51.279013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:51.278976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:53.146284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:26:53.146244 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:09.625381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:09.625345 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:10.302529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:10.302489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:10.432405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:10.432288 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:10.602691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:10.602607 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:15.937264 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:27:15.937226 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:27:28.223431 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:28.223377 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:27:32.893330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.893295 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn"] Oct 31 03:27:32.893756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.893703 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8869ad49-2264-4ddf-895b-9ca8e5a01b3c" containerName="revoke-keys" Oct 31 03:27:32.893756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.893717 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8869ad49-2264-4ddf-895b-9ca8e5a01b3c" containerName="revoke-keys" Oct 31 03:27:32.893837 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.893801 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8869ad49-2264-4ddf-895b-9ca8e5a01b3c" containerName="revoke-keys" Oct 31 03:27:32.896923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.896904 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:32.899733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.899708 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:27:32.900045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.900021 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:27:32.900146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.900126 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:27:32.900196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.900127 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:27:32.908174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.908139 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn"] Oct 31 03:27:32.978184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.978149 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:32.978379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.978274 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:32.978379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:32.978307 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nkb\" (UniqueName: \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.079616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.079578 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.079616 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.079622 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nkb\" (UniqueName: \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.079919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.079713 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.080342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.080315 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.082067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.082044 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.091629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.091602 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nkb\" (UniqueName: \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\") pod \"maas-api-revoke-sub-c1e271a9-mq2sn\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.212877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.212834 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:33.487722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.487694 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn"] Oct 31 03:27:33.490182 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:27:33.490146 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dba02ef_edbf_4e88_ad78_c2469a46de7b.slice/crio-13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf WatchSource:0}: Error finding container 13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf: Status 404 returned error can't find the container with id 13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf Oct 31 03:27:33.882276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.882242 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" event={"ID":"5dba02ef-edbf-4e88-ad78-c2469a46de7b","Type":"ContainerStarted","Data":"880ff203b9b8b8339b4be2a3654d56d31916577a1431c6707449ad97064069c5"} Oct 31 03:27:33.882486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.882295 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" event={"ID":"5dba02ef-edbf-4e88-ad78-c2469a46de7b","Type":"ContainerStarted","Data":"13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf"} Oct 31 03:27:33.901320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:33.901269 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" podStartSLOduration=1.901253583 podStartE2EDuration="1.901253583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:33.899741286 +0000 UTC m=+1931.754758428" watchObservedRunningTime="2026-10-31 03:27:33.901253583 +0000 UTC m=+1931.756270751" Oct 31 03:27:34.887897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:34.887862 4556 generic.go:356] "Generic (PLEG): container finished" podID="5dba02ef-edbf-4e88-ad78-c2469a46de7b" containerID="880ff203b9b8b8339b4be2a3654d56d31916577a1431c6707449ad97064069c5" exitCode=0 Oct 31 03:27:34.888092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:34.887949 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" event={"ID":"5dba02ef-edbf-4e88-ad78-c2469a46de7b","Type":"ContainerDied","Data":"880ff203b9b8b8339b4be2a3654d56d31916577a1431c6707449ad97064069c5"} Oct 31 03:27:36.032328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.032299 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:36.107027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.106990 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\") pod \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " Oct 31 03:27:36.107238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.107037 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\") pod \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " Oct 31 03:27:36.107238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.107126 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\" (UniqueName: \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\") pod \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\" (UID: \"5dba02ef-edbf-4e88-ad78-c2469a46de7b\") " Oct 31 03:27:36.107521 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.107494 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca" pod "5dba02ef-edbf-4e88-ad78-c2469a46de7b" (UID: "5dba02ef-edbf-4e88-ad78-c2469a46de7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:27:36.109242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.109212 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca" pod "5dba02ef-edbf-4e88-ad78-c2469a46de7b" (UID: "5dba02ef-edbf-4e88-ad78-c2469a46de7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:27:36.109356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.109335 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb" pod "5dba02ef-edbf-4e88-ad78-c2469a46de7b" (UID: "5dba02ef-edbf-4e88-ad78-c2469a46de7b"). InnerVolumeSpecName "kube-api-access-r4nkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:27:36.208821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.208784 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4nkb\" (UniqueName: \"kubernetes.io/projected/5dba02ef-edbf-4e88-ad78-c2469a46de7b-kube-api-access-r4nkb\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:36.208821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.208821 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:36.209040 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.208831 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dba02ef-edbf-4e88-ad78-c2469a46de7b-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:36.897582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.897554 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" Oct 31 03:27:36.897765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.897558 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn" event={"ID":"5dba02ef-edbf-4e88-ad78-c2469a46de7b","Type":"ContainerDied","Data":"13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf"} Oct 31 03:27:36.897765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:36.897689 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13aefc4e4792729873f5666a875318121c399ecbdb60d46806d1006b77548bcf" Oct 31 03:27:43.656790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:43.656752 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:27:45.942859 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:27:45.942813 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:27:47.411434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:47.410960 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:27:49.315227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:49.315188 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:49.447820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:49.447772 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:51.631514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.631479 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb"] Oct 31 03:27:51.631914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.631896 4556 cpu_manager.go:414] "containerMap: removing container" podUID="5dba02ef-edbf-4e88-ad78-c2469a46de7b" containerName="revoke-keys" Oct 31 03:27:51.631914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.631907 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dba02ef-edbf-4e88-ad78-c2469a46de7b" containerName="revoke-keys" Oct 31 03:27:51.631989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.631969 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dba02ef-edbf-4e88-ad78-c2469a46de7b" containerName="revoke-keys" Oct 31 03:27:51.635171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.635153 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.638165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.638139 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:27:51.638298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.638179 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:27:51.638367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.638334 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:27:51.638367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.638346 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:27:51.644465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.644438 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb"] Oct 31 03:27:51.796879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.796815 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.796879 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.796887 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.797115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.797041 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4npjm\" (UniqueName: \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.897856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.897753 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4npjm\" (UniqueName: \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.897856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.897849 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.898102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.897887 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.898659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.898630 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.900462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.900435 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.910709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.910685 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4npjm\" (UniqueName: \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\") pod \"maas-api-revoke-sub-2e1f5b5f-cfklb\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:51.948700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:51.948665 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:52.227853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:52.227818 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb"] Oct 31 03:27:52.232339 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:27:52.232310 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09945448_737d_4899_9e10_d391fb60fd7b.slice/crio-82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45 WatchSource:0}: Error finding container 82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45: Status 404 returned error can't find the container with id 82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45 Oct 31 03:27:52.982267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:52.982223 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" event={"ID":"09945448-737d-4899-9e10-d391fb60fd7b","Type":"ContainerStarted","Data":"ee07a1ea5ef6018fe25c20544dd1ffc3c638f72f789b2bf7899ba714d1eb0b76"} Oct 31 03:27:52.982762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:52.982295 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" event={"ID":"09945448-737d-4899-9e10-d391fb60fd7b","Type":"ContainerStarted","Data":"82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45"} Oct 31 03:27:53.000936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:53.000884 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" podStartSLOduration=2.000868471 podStartE2EDuration="2.000868471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:52.999094338 +0000 UTC m=+1950.854111478" watchObservedRunningTime="2026-10-31 03:27:53.000868471 +0000 UTC m=+1950.855885612" Oct 31 03:27:53.989676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:53.989641 4556 generic.go:356] "Generic (PLEG): container finished" podID="09945448-737d-4899-9e10-d391fb60fd7b" containerID="ee07a1ea5ef6018fe25c20544dd1ffc3c638f72f789b2bf7899ba714d1eb0b76" exitCode=0 Oct 31 03:27:53.990085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:53.989724 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" event={"ID":"09945448-737d-4899-9e10-d391fb60fd7b","Type":"ContainerDied","Data":"ee07a1ea5ef6018fe25c20544dd1ffc3c638f72f789b2bf7899ba714d1eb0b76"} Oct 31 03:27:55.137397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.137371 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:55.233005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.232968 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\") pod \"09945448-737d-4899-9e10-d391fb60fd7b\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " Oct 31 03:27:55.233005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.233009 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\" (UniqueName: \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\") pod \"09945448-737d-4899-9e10-d391fb60fd7b\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " Oct 31 03:27:55.233238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.233206 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\") pod \"09945448-737d-4899-9e10-d391fb60fd7b\" (UID: \"09945448-737d-4899-9e10-d391fb60fd7b\") " Oct 31 03:27:55.233614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.233577 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca" pod "09945448-737d-4899-9e10-d391fb60fd7b" (UID: "09945448-737d-4899-9e10-d391fb60fd7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:27:55.235234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.235211 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca" pod "09945448-737d-4899-9e10-d391fb60fd7b" (UID: "09945448-737d-4899-9e10-d391fb60fd7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:27:55.235322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.235218 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm" pod "09945448-737d-4899-9e10-d391fb60fd7b" (UID: "09945448-737d-4899-9e10-d391fb60fd7b"). InnerVolumeSpecName "kube-api-access-4npjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:27:55.335108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.335012 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:55.335108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.335052 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09945448-737d-4899-9e10-d391fb60fd7b-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:55.335108 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.335063 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4npjm\" (UniqueName: \"kubernetes.io/projected/09945448-737d-4899-9e10-d391fb60fd7b-kube-api-access-4npjm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:27:55.999991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.999956 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" Oct 31 03:27:56.000187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:55.999962 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb" event={"ID":"09945448-737d-4899-9e10-d391fb60fd7b","Type":"ContainerDied","Data":"82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45"} Oct 31 03:27:56.000187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:27:56.000080 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cf122ae0ce8d2efdcace2ccf84f5ab4fa3788ecbed487edc1d9434a3ddca45" Oct 31 03:28:02.454455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.454395 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:28:02.646640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.646603 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8"] Oct 31 03:28:02.646996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.646983 4556 cpu_manager.go:414] "containerMap: removing container" podUID="09945448-737d-4899-9e10-d391fb60fd7b" containerName="revoke-keys" Oct 31 03:28:02.647041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.646997 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09945448-737d-4899-9e10-d391fb60fd7b" containerName="revoke-keys" Oct 31 03:28:02.647128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.647119 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="09945448-737d-4899-9e10-d391fb60fd7b" containerName="revoke-keys" Oct 31 03:28:02.650453 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.650406 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.653394 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.653364 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:28:02.666746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.666654 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:28:02.667001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.666783 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:28:02.667131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.666816 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:28:02.672378 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.672340 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8"] Oct 31 03:28:02.710012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.709920 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.710012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.709998 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.710237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.710062 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnllg\" (UniqueName: \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.811400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.811338 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.811677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.811489 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.811677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.811573 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnllg\" (UniqueName: \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.812257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.812234 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.813969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.813944 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.823311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.823274 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnllg\" (UniqueName: \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\") pod \"maas-api-revoke-sub-ec843018-plfq8\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:02.980840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:02.980754 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:03.256642 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:03.256612 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8"] Oct 31 03:28:03.259167 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:28:03.259134 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c382ec_1eb7_4f74_ad97_7c75a0d7c763.slice/crio-35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f WatchSource:0}: Error finding container 35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f: Status 404 returned error can't find the container with id 35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f Oct 31 03:28:04.039032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:04.038992 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" event={"ID":"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763","Type":"ContainerStarted","Data":"1638f86ca21f4178a4efa9cc8e554d5d26a6ce010ee676c781e379baace8db0f"} Oct 31 03:28:04.039517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:04.039056 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" event={"ID":"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763","Type":"ContainerStarted","Data":"35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f"} Oct 31 03:28:04.059503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:04.059448 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" podStartSLOduration=2.059431728 podStartE2EDuration="2.059431728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:04.05752342 +0000 UTC m=+1961.912540562" watchObservedRunningTime="2026-10-31 03:28:04.059431728 +0000 UTC m=+1961.914448863" Oct 31 03:28:05.048112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:05.048055 4556 generic.go:356] "Generic (PLEG): container finished" podID="c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" containerID="1638f86ca21f4178a4efa9cc8e554d5d26a6ce010ee676c781e379baace8db0f" exitCode=0 Oct 31 03:28:05.048686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:05.048133 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" event={"ID":"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763","Type":"ContainerDied","Data":"1638f86ca21f4178a4efa9cc8e554d5d26a6ce010ee676c781e379baace8db0f"} Oct 31 03:28:06.452486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.451984 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:06.457881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.457323 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\") pod \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " Oct 31 03:28:06.457881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.457443 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\" (UniqueName: \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\") pod \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " Oct 31 03:28:06.457881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.457489 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\") pod \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\" (UID: \"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763\") " Oct 31 03:28:06.458183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.458070 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca" pod "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" (UID: "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:28:06.459783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.459753 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca" pod "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" (UID: "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:06.460041 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.460019 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg" pod "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" (UID: "c2c382ec-1eb7-4f74-ad97-7c75a0d7c763"). InnerVolumeSpecName "kube-api-access-rnllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:28:06.570441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.567222 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:06.570441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.567260 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnllg\" (UniqueName: \"kubernetes.io/projected/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-kube-api-access-rnllg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:06.570441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:06.567272 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:07.058980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:07.058949 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" Oct 31 03:28:07.059172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:07.058979 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8" event={"ID":"c2c382ec-1eb7-4f74-ad97-7c75a0d7c763","Type":"ContainerDied","Data":"35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f"} Oct 31 03:28:07.059172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:07.059035 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f4d7eb6c28c3b0042d6d9b8b60bdd2026cc3ba912715ff243f865e704dd83f" Oct 31 03:28:09.665487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:09.665437 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:10.432191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:10.432154 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:15.460289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:15.460248 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:28:15.941498 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:28:15.941454 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:28:17.731625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:17.731570 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:17.863441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:17.863375 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:29.062309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.062270 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5"] Oct 31 03:28:29.062770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.062680 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" containerName="revoke-keys" Oct 31 03:28:29.062770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.062693 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" containerName="revoke-keys" Oct 31 03:28:29.062770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.062771 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" containerName="revoke-keys" Oct 31 03:28:29.066253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.066229 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.082951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.070134 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:28:29.082951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.070241 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:28:29.082951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.070440 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:28:29.082951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.070567 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:28:29.085761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.085717 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5"] Oct 31 03:28:29.165842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.165794 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.166043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.165946 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.166089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.166044 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64dx\" (UniqueName: \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.267017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.266975 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64dx\" (UniqueName: \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.267265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.267077 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.267265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.267136 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.267931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.267909 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.269809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.269783 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.279940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.279904 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64dx\" (UniqueName: \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\") pod \"maas-api-revoke-sub-046e1343-tw2m5\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.398784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.398693 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:29.680801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:29.680746 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5"] Oct 31 03:28:29.683808 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:28:29.683776 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fb0b3c_bd36_4dff_a139_28868e5eff8b.slice/crio-6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6 WatchSource:0}: Error finding container 6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6: Status 404 returned error can't find the container with id 6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6 Oct 31 03:28:30.198103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:30.198064 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" event={"ID":"c5fb0b3c-bd36-4dff-a139-28868e5eff8b","Type":"ContainerStarted","Data":"1492edfb786526110dd19720db7daa94a2e59829e3d316303d58ee149c17d0f4"} Oct 31 03:28:30.198530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:30.198126 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" event={"ID":"c5fb0b3c-bd36-4dff-a139-28868e5eff8b","Type":"ContainerStarted","Data":"6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6"} Oct 31 03:28:30.217631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:30.217573 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" podStartSLOduration=1.217556009 podStartE2EDuration="1.217556009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:30.215207902 +0000 UTC m=+1988.070225044" watchObservedRunningTime="2026-10-31 03:28:30.217556009 +0000 UTC m=+1988.072573149" Oct 31 03:28:31.204816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:31.204773 4556 generic.go:356] "Generic (PLEG): container finished" podID="c5fb0b3c-bd36-4dff-a139-28868e5eff8b" containerID="1492edfb786526110dd19720db7daa94a2e59829e3d316303d58ee149c17d0f4" exitCode=0 Oct 31 03:28:31.205265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:31.204860 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" event={"ID":"c5fb0b3c-bd36-4dff-a139-28868e5eff8b","Type":"ContainerDied","Data":"1492edfb786526110dd19720db7daa94a2e59829e3d316303d58ee149c17d0f4"} Oct 31 03:28:32.351220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.351165 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:32.399244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.399205 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\") pod \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " Oct 31 03:28:32.399429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.399305 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\" (UniqueName: \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\") pod \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " Oct 31 03:28:32.399429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.399395 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\") pod \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\" (UID: \"c5fb0b3c-bd36-4dff-a139-28868e5eff8b\") " Oct 31 03:28:32.399844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.399810 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca" pod "c5fb0b3c-bd36-4dff-a139-28868e5eff8b" (UID: "c5fb0b3c-bd36-4dff-a139-28868e5eff8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:28:32.401555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.401530 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca" pod "c5fb0b3c-bd36-4dff-a139-28868e5eff8b" (UID: "c5fb0b3c-bd36-4dff-a139-28868e5eff8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:32.401639 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.401595 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx" pod "c5fb0b3c-bd36-4dff-a139-28868e5eff8b" (UID: "c5fb0b3c-bd36-4dff-a139-28868e5eff8b"). InnerVolumeSpecName "kube-api-access-q64dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:28:32.501452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.501322 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:32.501452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.501359 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:32.501452 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:32.501369 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q64dx\" (UniqueName: \"kubernetes.io/projected/c5fb0b3c-bd36-4dff-a139-28868e5eff8b-kube-api-access-q64dx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:33.215514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:33.215479 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" Oct 31 03:28:33.215514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:33.215492 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5" event={"ID":"c5fb0b3c-bd36-4dff-a139-28868e5eff8b","Type":"ContainerDied","Data":"6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6"} Oct 31 03:28:33.215756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:33.215573 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4849d06093e13274be7d39a62ea6c98b68c9ab9c2e0770cd6943fea2ecb6d6" Oct 31 03:28:39.299877 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.299834 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm"] Oct 31 03:28:39.300380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.300237 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c5fb0b3c-bd36-4dff-a139-28868e5eff8b" containerName="revoke-keys" Oct 31 03:28:39.300380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.300248 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fb0b3c-bd36-4dff-a139-28868e5eff8b" containerName="revoke-keys" Oct 31 03:28:39.300380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.300321 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fb0b3c-bd36-4dff-a139-28868e5eff8b" containerName="revoke-keys" Oct 31 03:28:39.303741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.303717 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.307397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.307367 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:28:39.307667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.307643 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:28:39.307797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.307778 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:28:39.307859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.307780 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:28:39.314916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.314880 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm"] Oct 31 03:28:39.372151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.372102 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.372345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.372176 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.372345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.372303 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48x4g\" (UniqueName: \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.473828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.473780 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.474055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.473858 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.474055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.473921 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48x4g\" (UniqueName: \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.474635 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.474615 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.476402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.476381 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.486307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.486276 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48x4g\" (UniqueName: \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\") pod \"maas-api-revoke-sub-d3fba056-lxvqm\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.619952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.619858 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:39.902627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.902593 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm"] Oct 31 03:28:39.904701 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:28:39.904667 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c3f3e5_105e_44d3_b106_4a9613b22a16.slice/crio-2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03 WatchSource:0}: Error finding container 2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03: Status 404 returned error can't find the container with id 2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03 Oct 31 03:28:39.926975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:39.926941 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:28:40.257401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:40.257364 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" event={"ID":"43c3f3e5-105e-44d3-b106-4a9613b22a16","Type":"ContainerStarted","Data":"9a3caad7f166f2f6fb03a5736f7ef701b96d9d5d21dccc258b8ae403c3a8694d"} Oct 31 03:28:40.257602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:40.257456 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" event={"ID":"43c3f3e5-105e-44d3-b106-4a9613b22a16","Type":"ContainerStarted","Data":"2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03"} Oct 31 03:28:41.259897 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:41.259857 4556 generic.go:356] "Generic (PLEG): container finished" podID="43c3f3e5-105e-44d3-b106-4a9613b22a16" containerID="9a3caad7f166f2f6fb03a5736f7ef701b96d9d5d21dccc258b8ae403c3a8694d" exitCode=0 Oct 31 03:28:41.260346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:41.259950 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" event={"ID":"43c3f3e5-105e-44d3-b106-4a9613b22a16","Type":"ContainerDied","Data":"9a3caad7f166f2f6fb03a5736f7ef701b96d9d5d21dccc258b8ae403c3a8694d"} Oct 31 03:28:42.642249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.642209 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:42.755090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.755056 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\" (UniqueName: \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\") pod \"43c3f3e5-105e-44d3-b106-4a9613b22a16\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " Oct 31 03:28:42.755337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.755116 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\") pod \"43c3f3e5-105e-44d3-b106-4a9613b22a16\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " Oct 31 03:28:42.755337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.755176 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\") pod \"43c3f3e5-105e-44d3-b106-4a9613b22a16\" (UID: \"43c3f3e5-105e-44d3-b106-4a9613b22a16\") " Oct 31 03:28:42.755641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.755615 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca" pod "43c3f3e5-105e-44d3-b106-4a9613b22a16" (UID: "43c3f3e5-105e-44d3-b106-4a9613b22a16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:28:42.757365 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.757329 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca" pod "43c3f3e5-105e-44d3-b106-4a9613b22a16" (UID: "43c3f3e5-105e-44d3-b106-4a9613b22a16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:28:42.757514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.757449 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g" pod "43c3f3e5-105e-44d3-b106-4a9613b22a16" (UID: "43c3f3e5-105e-44d3-b106-4a9613b22a16"). InnerVolumeSpecName "kube-api-access-48x4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:28:42.857049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.856934 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48x4g\" (UniqueName: \"kubernetes.io/projected/43c3f3e5-105e-44d3-b106-4a9613b22a16-kube-api-access-48x4g\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:42.857049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.856991 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:42.857049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:42.857008 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c3f3e5-105e-44d3-b106-4a9613b22a16-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:28:43.271632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:43.271599 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" Oct 31 03:28:43.271833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:43.271637 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm" event={"ID":"43c3f3e5-105e-44d3-b106-4a9613b22a16","Type":"ContainerDied","Data":"2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03"} Oct 31 03:28:43.271833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:43.271699 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2286caba9c50f253c7faf798addf6e14d8fbff5861be3978039031a158d59d03" Oct 31 03:28:45.936958 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:28:45.936916 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:28:50.134902 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:50.134859 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:28:55.261940 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:28:55.261892 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:00.293789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.293751 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8"] Oct 31 03:29:00.294214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.294159 4556 cpu_manager.go:414] "containerMap: removing container" podUID="43c3f3e5-105e-44d3-b106-4a9613b22a16" containerName="revoke-keys" Oct 31 03:29:00.294214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.294171 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c3f3e5-105e-44d3-b106-4a9613b22a16" containerName="revoke-keys" Oct 31 03:29:00.294295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.294232 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c3f3e5-105e-44d3-b106-4a9613b22a16" containerName="revoke-keys" Oct 31 03:29:00.297506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.297487 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.301719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.301698 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:29:00.301851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.301833 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:29:00.302182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.302161 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:29:00.302508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.302485 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:29:00.311306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.311277 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8"] Oct 31 03:29:00.469637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.469592 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.469829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.469649 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.469829 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.469800 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjgv\" (UniqueName: \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.570730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.570633 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjgv\" (UniqueName: \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.570730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.570723 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.570934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.570753 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.571478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.571444 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.573127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.573104 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.583240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.583210 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjgv\" (UniqueName: \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\") pod \"maas-api-revoke-sub-ae594979-4j8f8\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.611039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.610988 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:00.894927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:00.894892 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8"] Oct 31 03:29:00.896743 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:29:00.896708 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b5d86e_731e_4c17_a1cb_ef23a49da733.slice/crio-41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb WatchSource:0}: Error finding container 41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb: Status 404 returned error can't find the container with id 41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb Oct 31 03:29:01.382705 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:01.382671 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" event={"ID":"c2b5d86e-731e-4c17-a1cb-ef23a49da733","Type":"ContainerStarted","Data":"84f798981149fc60b4e55d63573479bce5a5ddfb46255d1c2155f6d29299114d"} Oct 31 03:29:01.383197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:01.382721 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" event={"ID":"c2b5d86e-731e-4c17-a1cb-ef23a49da733","Type":"ContainerStarted","Data":"41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb"} Oct 31 03:29:01.406362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:01.406301 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" podStartSLOduration=1.406285368 podStartE2EDuration="1.406285368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:01.403675811 +0000 UTC m=+2019.258692953" watchObservedRunningTime="2026-10-31 03:29:01.406285368 +0000 UTC m=+2019.261302552" Oct 31 03:29:02.392853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:02.392815 4556 generic.go:356] "Generic (PLEG): container finished" podID="c2b5d86e-731e-4c17-a1cb-ef23a49da733" containerID="84f798981149fc60b4e55d63573479bce5a5ddfb46255d1c2155f6d29299114d" exitCode=0 Oct 31 03:29:02.393266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:02.392870 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" event={"ID":"c2b5d86e-731e-4c17-a1cb-ef23a49da733","Type":"ContainerDied","Data":"84f798981149fc60b4e55d63573479bce5a5ddfb46255d1c2155f6d29299114d"} Oct 31 03:29:03.545482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.545453 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:03.705864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.705770 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\" (UniqueName: \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\") pod \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " Oct 31 03:29:03.705864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.705813 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\") pod \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " Oct 31 03:29:03.706141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.705869 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\") pod \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\" (UID: \"c2b5d86e-731e-4c17-a1cb-ef23a49da733\") " Oct 31 03:29:03.706256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.706225 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca" pod "c2b5d86e-731e-4c17-a1cb-ef23a49da733" (UID: "c2b5d86e-731e-4c17-a1cb-ef23a49da733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:03.707985 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.707958 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv" pod "c2b5d86e-731e-4c17-a1cb-ef23a49da733" (UID: "c2b5d86e-731e-4c17-a1cb-ef23a49da733"). InnerVolumeSpecName "kube-api-access-qnjgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:03.708110 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.708032 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca" pod "c2b5d86e-731e-4c17-a1cb-ef23a49da733" (UID: "c2b5d86e-731e-4c17-a1cb-ef23a49da733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:29:03.807560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.807520 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjgv\" (UniqueName: \"kubernetes.io/projected/c2b5d86e-731e-4c17-a1cb-ef23a49da733-kube-api-access-qnjgv\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:03.807560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.807553 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:03.807560 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:03.807567 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b5d86e-731e-4c17-a1cb-ef23a49da733-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:04.411355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:04.411322 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" Oct 31 03:29:04.411537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:04.411364 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8" event={"ID":"c2b5d86e-731e-4c17-a1cb-ef23a49da733","Type":"ContainerDied","Data":"41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb"} Oct 31 03:29:04.411537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:04.411445 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41143d75fa72ca3a04756dd3bf5319c92466e74ea816b7038922412cb0dc58bb" Oct 31 03:29:09.718512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:09.718477 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:10.484821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:10.484788 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:11.123995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:11.123957 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:14.858759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:14.858712 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:15.944401 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:29:15.944347 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:29:18.656021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.655976 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m"] Oct 31 03:29:18.656444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.656380 4556 cpu_manager.go:414] "containerMap: removing container" podUID="c2b5d86e-731e-4c17-a1cb-ef23a49da733" containerName="revoke-keys" Oct 31 03:29:18.656444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.656390 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b5d86e-731e-4c17-a1cb-ef23a49da733" containerName="revoke-keys" Oct 31 03:29:18.656531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.656466 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b5d86e-731e-4c17-a1cb-ef23a49da733" containerName="revoke-keys" Oct 31 03:29:18.659858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.659835 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.663683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.663640 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:29:18.663995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.663975 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:29:18.664100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.664006 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:29:18.664100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.664093 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:29:18.667536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.667510 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.667677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.667640 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9m6w\" (UniqueName: \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.667764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.667744 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.672637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.672610 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m"] Oct 31 03:29:18.768787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.768737 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9m6w\" (UniqueName: \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.768787 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.768790 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.769031 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.768845 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.769535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.769512 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.771331 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.771309 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.781251 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.781226 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9m6w\" (UniqueName: \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\") pod \"maas-api-revoke-sub-a4aef7d7-4227m\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:18.974101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:18.974012 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:19.248782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:19.248751 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m"] Oct 31 03:29:19.251684 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:29:19.251635 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54323503_6e68_4dbe_9874_622c2fe4afe6.slice/crio-7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36 WatchSource:0}: Error finding container 7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36: Status 404 returned error can't find the container with id 7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36 Oct 31 03:29:19.541688 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:19.541594 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" event={"ID":"54323503-6e68-4dbe-9874-622c2fe4afe6","Type":"ContainerStarted","Data":"7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36"} Oct 31 03:29:20.548042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:20.547949 4556 generic.go:356] "Generic (PLEG): container finished" podID="54323503-6e68-4dbe-9874-622c2fe4afe6" containerID="20c3399816a8239c40a49dcf3aaed6fedb661e3937ef2eca9f6d4774857ef433" exitCode=0 Oct 31 03:29:20.548042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:20.548029 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" event={"ID":"54323503-6e68-4dbe-9874-622c2fe4afe6","Type":"ContainerDied","Data":"20c3399816a8239c40a49dcf3aaed6fedb661e3937ef2eca9f6d4774857ef433"} Oct 31 03:29:21.699794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.699758 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:21.796150 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.796106 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\") pod \"54323503-6e68-4dbe-9874-622c2fe4afe6\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " Oct 31 03:29:21.796369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.796162 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\" (UniqueName: \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\") pod \"54323503-6e68-4dbe-9874-622c2fe4afe6\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " Oct 31 03:29:21.796369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.796210 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\") pod \"54323503-6e68-4dbe-9874-622c2fe4afe6\" (UID: \"54323503-6e68-4dbe-9874-622c2fe4afe6\") " Oct 31 03:29:21.796618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.796593 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca" pod "54323503-6e68-4dbe-9874-622c2fe4afe6" (UID: "54323503-6e68-4dbe-9874-622c2fe4afe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:21.798397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.798374 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca" pod "54323503-6e68-4dbe-9874-622c2fe4afe6" (UID: "54323503-6e68-4dbe-9874-622c2fe4afe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:29:21.798504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.798453 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w" pod "54323503-6e68-4dbe-9874-622c2fe4afe6" (UID: "54323503-6e68-4dbe-9874-622c2fe4afe6"). InnerVolumeSpecName "kube-api-access-g9m6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:21.897881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.897836 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:21.897881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.897874 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9m6w\" (UniqueName: \"kubernetes.io/projected/54323503-6e68-4dbe-9874-622c2fe4afe6-kube-api-access-g9m6w\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:21.897881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:21.897887 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54323503-6e68-4dbe-9874-622c2fe4afe6-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:22.559800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:22.559720 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" Oct 31 03:29:22.559800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:22.559743 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m" event={"ID":"54323503-6e68-4dbe-9874-622c2fe4afe6","Type":"ContainerDied","Data":"7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36"} Oct 31 03:29:22.559800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:22.559791 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2f975645debb89e84c989729d3aa646ee9889a0df4912f0e989a7633014b36" Oct 31 03:29:29.525917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:29.525873 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:33.875328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:33.874211 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:37.506383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.506342 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn"] Oct 31 03:29:37.506784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.506757 4556 cpu_manager.go:414] "containerMap: removing container" podUID="54323503-6e68-4dbe-9874-622c2fe4afe6" containerName="revoke-keys" Oct 31 03:29:37.506784 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.506769 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54323503-6e68-4dbe-9874-622c2fe4afe6" containerName="revoke-keys" Oct 31 03:29:37.506885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.506846 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="54323503-6e68-4dbe-9874-622c2fe4afe6" containerName="revoke-keys" Oct 31 03:29:37.511310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.511282 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.517030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.517000 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:29:37.517238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.517105 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:29:37.517322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.517105 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:29:37.517322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.517126 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:29:37.520903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.520866 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn"] Oct 31 03:29:37.591519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.591482 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.591519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.591533 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.591793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.591555 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdks7\" (UniqueName: \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.692739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.692684 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdks7\" (UniqueName: \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.692923 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.692891 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.692964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.692940 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.693764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.693737 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.695484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.695455 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.704970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.704942 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdks7\" (UniqueName: \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\") pod \"maas-api-revoke-sub-79805ca9-896gn\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:37.830117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:37.830026 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:38.108483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:38.108308 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn"] Oct 31 03:29:38.111435 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:29:38.111373 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b45e4e_5e1f_4539_a6fb_bdb4b4bda96e.slice/crio-42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe WatchSource:0}: Error finding container 42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe: Status 404 returned error can't find the container with id 42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe Oct 31 03:29:38.673336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:38.673296 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" event={"ID":"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e","Type":"ContainerStarted","Data":"2de15c1372112ea50bd8b2123431801e8f056678d6e3ebf5043adc48aa3011a8"} Oct 31 03:29:38.673752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:38.673381 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" event={"ID":"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e","Type":"ContainerStarted","Data":"42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe"} Oct 31 03:29:38.691906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:38.691851 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" podStartSLOduration=1.691835188 podStartE2EDuration="1.691835188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:38.690118217 +0000 UTC m=+2056.545135359" watchObservedRunningTime="2026-10-31 03:29:38.691835188 +0000 UTC m=+2056.546852328" Oct 31 03:29:39.679782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:39.679744 4556 generic.go:356] "Generic (PLEG): container finished" podID="67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" containerID="2de15c1372112ea50bd8b2123431801e8f056678d6e3ebf5043adc48aa3011a8" exitCode=0 Oct 31 03:29:39.680254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:39.679828 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" event={"ID":"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e","Type":"ContainerDied","Data":"2de15c1372112ea50bd8b2123431801e8f056678d6e3ebf5043adc48aa3011a8"} Oct 31 03:29:40.824948 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.824919 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:40.927530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.927488 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\") pod \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " Oct 31 03:29:40.927748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.927585 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\" (UniqueName: \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\") pod \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " Oct 31 03:29:40.927748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.927701 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\") pod \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\" (UID: \"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e\") " Oct 31 03:29:40.928075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.928052 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca" pod "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" (UID: "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:40.929716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.929692 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca" pod "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" (UID: "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:29:40.929794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:40.929747 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7" pod "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" (UID: "67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e"). InnerVolumeSpecName "kube-api-access-sdks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:41.028589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.028544 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:41.028589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.028582 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdks7\" (UniqueName: \"kubernetes.io/projected/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-kube-api-access-sdks7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:41.028589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.028594 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:41.690482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.690448 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" Oct 31 03:29:41.690752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.690444 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn" event={"ID":"67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e","Type":"ContainerDied","Data":"42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe"} Oct 31 03:29:41.690752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:41.690588 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a248fd1b51c31ac54ac6621cce7aecce0b1dfcfbba64e646b70efb8fd381fe" Oct 31 03:29:45.937161 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:29:45.937118 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:29:48.343080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:48.343030 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:52.258619 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:52.258582 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:29:55.358584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.358545 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz"] Oct 31 03:29:55.358998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.358954 4556 cpu_manager.go:414] "containerMap: removing container" podUID="67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" containerName="revoke-keys" Oct 31 03:29:55.358998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.358965 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" containerName="revoke-keys" Oct 31 03:29:55.359075 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.359022 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" containerName="revoke-keys" Oct 31 03:29:55.362544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.362526 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.365652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.365620 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:29:55.365652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.365631 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:29:55.365833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.365631 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:29:55.365833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.365658 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:29:55.372713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.372681 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz"] Oct 31 03:29:55.483298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.483252 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpdr\" (UniqueName: \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.483544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.483393 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.483544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.483479 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.584380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.584333 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.584380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.584389 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.584640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.584489 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpdr\" (UniqueName: \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.585134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.585108 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.586858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.586833 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.596463 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.596435 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpdr\" (UniqueName: \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\") pod \"maas-api-revoke-sub-0b7e8edb-nqxnz\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.677467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.677433 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:55.956131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:55.955959 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz"] Oct 31 03:29:55.958462 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:29:55.958434 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8e35c1_64ae_4232_a740_fc86722741ba.slice/crio-a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d WatchSource:0}: Error finding container a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d: Status 404 returned error can't find the container with id a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d Oct 31 03:29:56.773066 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:56.773025 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" event={"ID":"6c8e35c1-64ae-4232-a740-fc86722741ba","Type":"ContainerStarted","Data":"ab96cdfb69d0b6ccdffa062c2a95c27f5b4ef637612a7592cf6539d337f3724e"} Oct 31 03:29:56.773584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:56.773131 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" event={"ID":"6c8e35c1-64ae-4232-a740-fc86722741ba","Type":"ContainerStarted","Data":"a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d"} Oct 31 03:29:56.790569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:56.790512 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" podStartSLOduration=1.790494046 podStartE2EDuration="1.790494046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:56.788892859 +0000 UTC m=+2074.643910001" watchObservedRunningTime="2026-10-31 03:29:56.790494046 +0000 UTC m=+2074.645511188" Oct 31 03:29:57.779120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:57.779071 4556 generic.go:356] "Generic (PLEG): container finished" podID="6c8e35c1-64ae-4232-a740-fc86722741ba" containerID="ab96cdfb69d0b6ccdffa062c2a95c27f5b4ef637612a7592cf6539d337f3724e" exitCode=0 Oct 31 03:29:57.779561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:57.779144 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" event={"ID":"6c8e35c1-64ae-4232-a740-fc86722741ba","Type":"ContainerDied","Data":"ab96cdfb69d0b6ccdffa062c2a95c27f5b4ef637612a7592cf6539d337f3724e"} Oct 31 03:29:58.919500 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:58.919463 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:59.020901 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.020860 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\" (UniqueName: \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\") pod \"6c8e35c1-64ae-4232-a740-fc86722741ba\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " Oct 31 03:29:59.021111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.020952 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\") pod \"6c8e35c1-64ae-4232-a740-fc86722741ba\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " Oct 31 03:29:59.021111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.020986 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\") pod \"6c8e35c1-64ae-4232-a740-fc86722741ba\" (UID: \"6c8e35c1-64ae-4232-a740-fc86722741ba\") " Oct 31 03:29:59.021315 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.021288 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca" pod "6c8e35c1-64ae-4232-a740-fc86722741ba" (UID: "6c8e35c1-64ae-4232-a740-fc86722741ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:29:59.023170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.023140 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca" pod "6c8e35c1-64ae-4232-a740-fc86722741ba" (UID: "6c8e35c1-64ae-4232-a740-fc86722741ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:29:59.023268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.023144 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr" pod "6c8e35c1-64ae-4232-a740-fc86722741ba" (UID: "6c8e35c1-64ae-4232-a740-fc86722741ba"). InnerVolumeSpecName "kube-api-access-bbpdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:29:59.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.122859 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpdr\" (UniqueName: \"kubernetes.io/projected/6c8e35c1-64ae-4232-a740-fc86722741ba-kube-api-access-bbpdr\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:59.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.122895 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:59.122951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.122907 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c8e35c1-64ae-4232-a740-fc86722741ba-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:29:59.791147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.791117 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" Oct 31 03:29:59.791334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.791152 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz" event={"ID":"6c8e35c1-64ae-4232-a740-fc86722741ba","Type":"ContainerDied","Data":"a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d"} Oct 31 03:29:59.791334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:29:59.791203 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a758fd0dc55269c181dfc1a5879a9e1f00370b2d285650355ac3b8332fc7142d" Oct 31 03:30:00.135566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.135485 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:00.135951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.135877 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6c8e35c1-64ae-4232-a740-fc86722741ba" containerName="revoke-keys" Oct 31 03:30:00.135951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.135888 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8e35c1-64ae-4232-a740-fc86722741ba" containerName="revoke-keys" Oct 31 03:30:00.136030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.135971 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8e35c1-64ae-4232-a740-fc86722741ba" containerName="revoke-keys" Oct 31 03:30:00.139124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.139105 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.142363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.142334 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:30:00.142545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.142459 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:30:00.142545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.142484 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:30:00.148014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.147981 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:00.236190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.236153 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.236376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.236204 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzm7d\" (UniqueName: \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.337117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.337075 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.337346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.337156 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzm7d\" (UniqueName: \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.337856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.337834 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.348972 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.348944 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzm7d\" (UniqueName: \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\") pod \"maas-api-key-cleanup-29890290-xsrcr\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.457131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.457085 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:00.732245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.732212 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:00.734691 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:30:00.734664 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e7d5b1_acea_4c68_bd31_d305a71c6371.slice/crio-eb58e6ad416f6137b6b4256930fffeb4f02c59013d323566e9c393f2173af23e WatchSource:0}: Error finding container eb58e6ad416f6137b6b4256930fffeb4f02c59013d323566e9c393f2173af23e: Status 404 returned error can't find the container with id eb58e6ad416f6137b6b4256930fffeb4f02c59013d323566e9c393f2173af23e Oct 31 03:30:00.797458 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:00.797391 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerStarted","Data":"eb58e6ad416f6137b6b4256930fffeb4f02c59013d323566e9c393f2173af23e"} Oct 31 03:30:01.804339 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:01.804301 4556 generic.go:356] "Generic (PLEG): container finished" podID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerID="69eb4be6c3385337e5cd91c1997c68dee0920b627fe18b9c3f9a7840b1f131e7" exitCode=60 Oct 31 03:30:01.804772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:01.804389 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerDied","Data":"69eb4be6c3385337e5cd91c1997c68dee0920b627fe18b9c3f9a7840b1f131e7"} Oct 31 03:30:01.804772 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:01.804611 4556 scope.go:122] "RemoveContainer" containerID="69eb4be6c3385337e5cd91c1997c68dee0920b627fe18b9c3f9a7840b1f131e7" Oct 31 03:30:02.810794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:02.810754 4556 generic.go:356] "Generic (PLEG): container finished" podID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerID="03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a" exitCode=60 Oct 31 03:30:02.811253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:02.810840 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerDied","Data":"03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a"} Oct 31 03:30:02.811253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:02.810897 4556 scope.go:122] "RemoveContainer" containerID="69eb4be6c3385337e5cd91c1997c68dee0920b627fe18b9c3f9a7840b1f131e7" Oct 31 03:30:02.811253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:02.811148 4556 scope.go:122] "RemoveContainer" containerID="03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a" Oct 31 03:30:02.811457 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:02.811437 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-xsrcr_odh-ai-gateway-infra(b4e7d5b1-acea-4c68-bd31-d305a71c6371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" Oct 31 03:30:03.818296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:03.818250 4556 scope.go:122] "RemoveContainer" containerID="03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a" Oct 31 03:30:03.818797 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:03.818588 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-xsrcr_odh-ai-gateway-infra(b4e7d5b1-acea-4c68-bd31-d305a71c6371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" Oct 31 03:30:06.230976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:06.230923 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:09.765904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:09.765873 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:10.558010 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:10.557976 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:13.250209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:13.250175 4556 scope.go:122] "RemoveContainer" containerID="03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a" Oct 31 03:30:13.869921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:13.869804 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerStarted","Data":"d8754ac54a4758aa5dc195576fcb3ef332f4a7bebb46fe70b898ebe95729c007"} Oct 31 03:30:13.890253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:13.890185 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" podStartSLOduration=13.890167778 podStartE2EDuration="13.890167778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:13.889786578 +0000 UTC m=+2091.744803743" watchObservedRunningTime="2026-10-31 03:30:13.890167778 +0000 UTC m=+2091.745184920" Oct 31 03:30:14.876747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:14.876707 4556 generic.go:356] "Generic (PLEG): container finished" podID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerID="d8754ac54a4758aa5dc195576fcb3ef332f4a7bebb46fe70b898ebe95729c007" exitCode=60 Oct 31 03:30:14.877184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:14.876776 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerDied","Data":"d8754ac54a4758aa5dc195576fcb3ef332f4a7bebb46fe70b898ebe95729c007"} Oct 31 03:30:14.877184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:14.876863 4556 scope.go:122] "RemoveContainer" containerID="03e7a51051c501b073c204fc92bae7b97920c90da546998cc0787867fd387d4a" Oct 31 03:30:14.877296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:14.877281 4556 scope.go:122] "RemoveContainer" containerID="d8754ac54a4758aa5dc195576fcb3ef332f4a7bebb46fe70b898ebe95729c007" Oct 31 03:30:14.877610 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:14.877590 4556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-xsrcr_odh-ai-gateway-infra(b4e7d5b1-acea-4c68-bd31-d305a71c6371)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" Oct 31 03:30:14.903067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:14.903030 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:15.940021 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:15.939983 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:30:16.027151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.027123 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:16.099090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.099050 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\") pod \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " Oct 31 03:30:16.099267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.099166 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\" (UniqueName: \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\") pod \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\" (UID: \"b4e7d5b1-acea-4c68-bd31-d305a71c6371\") " Oct 31 03:30:16.099538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.099513 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca" pod "b4e7d5b1-acea-4c68-bd31-d305a71c6371" (UID: "b4e7d5b1-acea-4c68-bd31-d305a71c6371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:16.101587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.101550 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d" pod "b4e7d5b1-acea-4c68-bd31-d305a71c6371" (UID: "b4e7d5b1-acea-4c68-bd31-d305a71c6371"). InnerVolumeSpecName "kube-api-access-mzm7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:16.200690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.200645 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7d5b1-acea-4c68-bd31-d305a71c6371-maas-api-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:16.200690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.200681 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzm7d\" (UniqueName: \"kubernetes.io/projected/b4e7d5b1-acea-4c68-bd31-d305a71c6371-kube-api-access-mzm7d\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:16.890078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.890050 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" Oct 31 03:30:16.890270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.890049 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr" event={"ID":"b4e7d5b1-acea-4c68-bd31-d305a71c6371","Type":"ContainerDied","Data":"eb58e6ad416f6137b6b4256930fffeb4f02c59013d323566e9c393f2173af23e"} Oct 31 03:30:16.890270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.890194 4556 scope.go:122] "RemoveContainer" containerID="d8754ac54a4758aa5dc195576fcb3ef332f4a7bebb46fe70b898ebe95729c007" Oct 31 03:30:16.906957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.906918 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:16.914753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.914719 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-xsrcr"] Oct 31 03:30:16.959162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:16.959122 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:18.269440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:18.267069 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" path="/var/lib/kubelet/pods/b4e7d5b1-acea-4c68-bd31-d305a71c6371/volumes" Oct 31 03:30:21.434397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:21.434351 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:24.427278 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.427241 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs"] Oct 31 03:30:24.428496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428450 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.428626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428499 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.428626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428519 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.428626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428527 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.428626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428547 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.428626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.428555 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.429080 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.429065 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.429139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.429087 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.429139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.429103 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e7d5b1-acea-4c68-bd31-d305a71c6371" containerName="cleanup" Oct 31 03:30:24.435889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.435855 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.439180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.439151 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:30:24.439455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.439276 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:30:24.439584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.439343 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:30:24.439795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.439763 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:30:24.441248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.441220 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs"] Oct 31 03:30:24.559106 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.559053 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.559286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.559201 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.559353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.559331 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscbd\" (UniqueName: \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.660306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.660265 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscbd\" (UniqueName: \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.660527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.660324 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.660527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.660383 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.661221 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.661187 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.662951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.662925 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.672533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.672504 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscbd\" (UniqueName: \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\") pod \"maas-api-revoke-sub-3d4156f1-7lvxs\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:24.753085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:24.752988 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:25.030079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:25.030049 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs"] Oct 31 03:30:25.032655 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:30:25.032619 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5091092e_0847_4706_8483_e44c303f93d3.slice/crio-580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4 WatchSource:0}: Error finding container 580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4: Status 404 returned error can't find the container with id 580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4 Oct 31 03:30:25.964551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:25.964449 4556 generic.go:356] "Generic (PLEG): container finished" podID="5091092e-0847-4706-8483-e44c303f93d3" containerID="3b25864ac61b465700798d8e4b2cc17cdec135eda61e8c8518ba9f08996c47c3" exitCode=0 Oct 31 03:30:25.964551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:25.964508 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" event={"ID":"5091092e-0847-4706-8483-e44c303f93d3","Type":"ContainerDied","Data":"3b25864ac61b465700798d8e4b2cc17cdec135eda61e8c8518ba9f08996c47c3"} Oct 31 03:30:25.964989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:25.964578 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" event={"ID":"5091092e-0847-4706-8483-e44c303f93d3","Type":"ContainerStarted","Data":"580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4"} Oct 31 03:30:27.104145 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.104119 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:27.290067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.289969 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\") pod \"5091092e-0847-4706-8483-e44c303f93d3\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " Oct 31 03:30:27.290067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.290028 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\" (UniqueName: \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\") pod \"5091092e-0847-4706-8483-e44c303f93d3\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " Oct 31 03:30:27.290321 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.290116 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\") pod \"5091092e-0847-4706-8483-e44c303f93d3\" (UID: \"5091092e-0847-4706-8483-e44c303f93d3\") " Oct 31 03:30:27.290550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.290527 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca" pod "5091092e-0847-4706-8483-e44c303f93d3" (UID: "5091092e-0847-4706-8483-e44c303f93d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:27.292197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.292170 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca" pod "5091092e-0847-4706-8483-e44c303f93d3" (UID: "5091092e-0847-4706-8483-e44c303f93d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:27.292287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.292266 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd" pod "5091092e-0847-4706-8483-e44c303f93d3" (UID: "5091092e-0847-4706-8483-e44c303f93d3"). InnerVolumeSpecName "kube-api-access-hscbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:27.391360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.391314 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5091092e-0847-4706-8483-e44c303f93d3-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:27.391360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.391355 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5091092e-0847-4706-8483-e44c303f93d3-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:27.391360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.391366 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscbd\" (UniqueName: \"kubernetes.io/projected/5091092e-0847-4706-8483-e44c303f93d3-kube-api-access-hscbd\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:27.977135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.977091 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" Oct 31 03:30:27.977135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.977120 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs" event={"ID":"5091092e-0847-4706-8483-e44c303f93d3","Type":"ContainerDied","Data":"580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4"} Oct 31 03:30:27.977398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:27.977166 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580f49f18efa9f686159deed129ea680ff75fcdff3ba5b21e3db2fd69458ecd4" Oct 31 03:30:28.900712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:28.900671 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:30:28.901220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:28.900971 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" podUID="d6d0b005-0520-436e-960d-261865375324" containerName="discovery" containerID="cri-o://3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa" gracePeriod=30 Oct 31 03:30:29.162586 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.162558 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:30:29.310198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310102 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310165 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310198 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310188 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310287 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310342 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310392 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310488 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\") pod \"d6d0b005-0520-436e-960d-261865375324\" (UID: \"d6d0b005-0520-436e-960d-261865375324\") " Oct 31 03:30:29.310773 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310694 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:29.310976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.310955 4556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6d0b005-0520-436e-960d-261865375324-istio-csr-ca-configmap\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.313216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313173 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:29.313333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313227 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:29.313333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313244 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:30:29.313333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313249 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:29.313333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313254 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:29.313333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.313267 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx" pod "d6d0b005-0520-436e-960d-261865375324" (UID: "d6d0b005-0520-436e-960d-261865375324"). InnerVolumeSpecName "kube-api-access-qcswx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:29.411890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411846 4556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-kubeconfig\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.411890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411888 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcswx\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-kube-api-access-qcswx\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.412133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411903 4556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-cacerts\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.412133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411917 4556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0b005-0520-436e-960d-261865375324-local-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.412133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411933 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0b005-0520-436e-960d-261865375324-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.412133 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.411949 4556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6d0b005-0520-436e-960d-261865375324-istio-csr-dns-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.987530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.987489 4556 generic.go:356] "Generic (PLEG): container finished" podID="d6d0b005-0520-436e-960d-261865375324" containerID="3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa" exitCode=0 Oct 31 03:30:29.988055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.987536 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" event={"ID":"d6d0b005-0520-436e-960d-261865375324","Type":"ContainerDied","Data":"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa"} Oct 31 03:30:29.988055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.987623 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" event={"ID":"d6d0b005-0520-436e-960d-261865375324","Type":"ContainerDied","Data":"cf36417918a010be0d4f4c02c8b80c7ad7332c9275ba129add1af15406702edd"} Oct 31 03:30:29.988055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.987553 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh" Oct 31 03:30:29.988055 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.987644 4556 scope.go:122] "RemoveContainer" containerID="3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa" Oct 31 03:30:29.998118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.998093 4556 scope.go:122] "RemoveContainer" containerID="3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa" Oct 31 03:30:29.998434 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:29.998397 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa\": container with ID starting with 3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa not found: ID does not exist" containerID="3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa" Oct 31 03:30:29.998525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:29.998446 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa"} err="failed to get container status \"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa\": rpc error: code = NotFound desc = could not find container \"3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa\": container with ID starting with 3799e88b8a3b0f7e36b9c3d7a51e4a3b9b9bd861eabcdec79762feb240b2befa not found: ID does not exist" Oct 31 03:30:30.020390 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:30.020353 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:30:30.029952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:30.029922 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-442jh"] Oct 31 03:30:30.256167 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:30.256087 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d0b005-0520-436e-960d-261865375324" path="/var/lib/kubelet/pods/d6d0b005-0520-436e-960d-261865375324/volumes" Oct 31 03:30:34.913871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:34.913791 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:35.243904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:35.243819 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:36.129095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:36.129055 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:38.547577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:38.547533 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:45.936273 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:30:45.936230 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:30:45.944012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:45.943982 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:30:53.529117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529081 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6"] Oct 31 03:30:53.529584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529495 4556 cpu_manager.go:414] "containerMap: removing container" podUID="d6d0b005-0520-436e-960d-261865375324" containerName="discovery" Oct 31 03:30:53.529584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529508 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d0b005-0520-436e-960d-261865375324" containerName="discovery" Oct 31 03:30:53.529584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529529 4556 cpu_manager.go:414] "containerMap: removing container" podUID="5091092e-0847-4706-8483-e44c303f93d3" containerName="revoke-keys" Oct 31 03:30:53.529584 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529534 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5091092e-0847-4706-8483-e44c303f93d3" containerName="revoke-keys" Oct 31 03:30:53.529750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529603 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d0b005-0520-436e-960d-261865375324" containerName="discovery" Oct 31 03:30:53.529750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.529614 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="5091092e-0847-4706-8483-e44c303f93d3" containerName="revoke-keys" Oct 31 03:30:53.532878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.532845 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.535888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.535861 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:30:53.536123 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.536105 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:30:53.536194 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.536158 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:30:53.536234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.536193 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:30:53.543949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.543905 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6"] Oct 31 03:30:53.588219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.588170 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.588489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.588292 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.588489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.588347 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrgh\" (UniqueName: \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.689835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.689789 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.690058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.689968 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.690058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.690036 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrgh\" (UniqueName: \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.690656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.690635 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.692277 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.692257 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.701404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.701372 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrgh\" (UniqueName: \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\") pod \"maas-api-revoke-sub-e2fc9c5b-4zzh6\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:53.850140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:53.850054 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:54.125539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:54.125505 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6"] Oct 31 03:30:54.128646 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:30:54.128615 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b635038_1e88_4f65_9a72_4df732131665.slice/crio-925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68 WatchSource:0}: Error finding container 925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68: Status 404 returned error can't find the container with id 925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68 Oct 31 03:30:54.140018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:54.139975 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" event={"ID":"8b635038-1e88-4f65-9a72-4df732131665","Type":"ContainerStarted","Data":"925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68"} Oct 31 03:30:55.025550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:55.025515 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz"] Oct 31 03:30:55.027721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:55.027690 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa83ed4-fsztz"] Oct 31 03:30:55.146382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:55.146303 4556 generic.go:356] "Generic (PLEG): container finished" podID="8b635038-1e88-4f65-9a72-4df732131665" containerID="4911bfed025824a7d0e65558e600c95e1d5f44261e6956513ddbbe833145d709" exitCode=0 Oct 31 03:30:55.146382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:55.146353 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" event={"ID":"8b635038-1e88-4f65-9a72-4df732131665","Type":"ContainerDied","Data":"4911bfed025824a7d0e65558e600c95e1d5f44261e6956513ddbbe833145d709"} Oct 31 03:30:56.255994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.255960 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6fd3aa-37ee-4806-96b3-e46df3a96b22" path="/var/lib/kubelet/pods/2c6fd3aa-37ee-4806-96b3-e46df3a96b22/volumes" Oct 31 03:30:56.289578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.289544 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:56.417257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.417163 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\") pod \"8b635038-1e88-4f65-9a72-4df732131665\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " Oct 31 03:30:56.417257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.417227 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\" (UniqueName: \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\") pod \"8b635038-1e88-4f65-9a72-4df732131665\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " Oct 31 03:30:56.417545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.417388 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\") pod \"8b635038-1e88-4f65-9a72-4df732131665\" (UID: \"8b635038-1e88-4f65-9a72-4df732131665\") " Oct 31 03:30:56.417781 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.417753 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca" pod "8b635038-1e88-4f65-9a72-4df732131665" (UID: "8b635038-1e88-4f65-9a72-4df732131665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:56.419569 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.419535 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca" pod "8b635038-1e88-4f65-9a72-4df732131665" (UID: "8b635038-1e88-4f65-9a72-4df732131665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:56.419709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.419657 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh" pod "8b635038-1e88-4f65-9a72-4df732131665" (UID: "8b635038-1e88-4f65-9a72-4df732131665"). InnerVolumeSpecName "kube-api-access-2lrgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:56.518930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.518889 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b635038-1e88-4f65-9a72-4df732131665-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:56.518930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.518925 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b635038-1e88-4f65-9a72-4df732131665-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:56.518930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:56.518938 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrgh\" (UniqueName: \"kubernetes.io/projected/8b635038-1e88-4f65-9a72-4df732131665-kube-api-access-2lrgh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:57.157643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:57.157615 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" Oct 31 03:30:57.157822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:57.157605 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6" event={"ID":"8b635038-1e88-4f65-9a72-4df732131665","Type":"ContainerDied","Data":"925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68"} Oct 31 03:30:57.157822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:30:57.157740 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925e38bb901b81f911c68a741ea0026d10d7ac0e43a1c6f7ce35639c980c9e68" Oct 31 03:31:03.777153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.777109 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56"] Oct 31 03:31:03.777726 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.777701 4556 cpu_manager.go:414] "containerMap: removing container" podUID="8b635038-1e88-4f65-9a72-4df732131665" containerName="revoke-keys" Oct 31 03:31:03.777810 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.777730 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b635038-1e88-4f65-9a72-4df732131665" containerName="revoke-keys" Oct 31 03:31:03.777890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.777877 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b635038-1e88-4f65-9a72-4df732131665" containerName="revoke-keys" Oct 31 03:31:03.781346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.781322 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.784363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.784334 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:31:03.784603 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.784583 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:31:03.784692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.784623 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:31:03.784692 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.784675 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:31:03.791818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.791774 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56"] Oct 31 03:31:03.890890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.890851 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.890890 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.890897 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpbm\" (UniqueName: \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.891148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.891036 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.991788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.991747 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.991991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.991802 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpbm\" (UniqueName: \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.991991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.991872 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.992543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.992521 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:03.994234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:03.994208 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:04.003525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:04.003499 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpbm\" (UniqueName: \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\") pod \"maas-api-revoke-sub-5a29efb3-2pc56\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:04.097684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:04.097580 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:04.351215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:04.351127 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:31:04.373932 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:04.373901 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56"] Oct 31 03:31:04.375568 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:31:04.375543 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e58b58a_3c6f_4499_a885_a8f8e953844b.slice/crio-838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83 WatchSource:0}: Error finding container 838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83: Status 404 returned error can't find the container with id 838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83 Oct 31 03:31:05.202875 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:05.202837 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" event={"ID":"3e58b58a-3c6f-4499-a885-a8f8e953844b","Type":"ContainerStarted","Data":"f8638d5333224cd68bd1abe9393cefb1224114345100ff8995b6961ee983dc59"} Oct 31 03:31:05.203457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:05.202904 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" event={"ID":"3e58b58a-3c6f-4499-a885-a8f8e953844b","Type":"ContainerStarted","Data":"838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83"} Oct 31 03:31:05.219434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:05.219367 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" podStartSLOduration=2.219351226 podStartE2EDuration="2.219351226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:05.219028831 +0000 UTC m=+2143.074045973" watchObservedRunningTime="2026-10-31 03:31:05.219351226 +0000 UTC m=+2143.074368367" Oct 31 03:31:06.240506 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:06.240466 4556 generic.go:356] "Generic (PLEG): container finished" podID="3e58b58a-3c6f-4499-a885-a8f8e953844b" containerID="f8638d5333224cd68bd1abe9393cefb1224114345100ff8995b6961ee983dc59" exitCode=0 Oct 31 03:31:06.241192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:06.240938 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" event={"ID":"3e58b58a-3c6f-4499-a885-a8f8e953844b","Type":"ContainerDied","Data":"f8638d5333224cd68bd1abe9393cefb1224114345100ff8995b6961ee983dc59"} Oct 31 03:31:07.423067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.423035 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:07.458334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.458294 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\") pod \"3e58b58a-3c6f-4499-a885-a8f8e953844b\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " Oct 31 03:31:07.458555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.458343 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\" (UniqueName: \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\") pod \"3e58b58a-3c6f-4499-a885-a8f8e953844b\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " Oct 31 03:31:07.458555 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.458450 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\") pod \"3e58b58a-3c6f-4499-a885-a8f8e953844b\" (UID: \"3e58b58a-3c6f-4499-a885-a8f8e953844b\") " Oct 31 03:31:07.458835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.458811 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca" pod "3e58b58a-3c6f-4499-a885-a8f8e953844b" (UID: "3e58b58a-3c6f-4499-a885-a8f8e953844b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:31:07.460615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.460589 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca" pod "3e58b58a-3c6f-4499-a885-a8f8e953844b" (UID: "3e58b58a-3c6f-4499-a885-a8f8e953844b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:31:07.460615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.460599 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm" pod "3e58b58a-3c6f-4499-a885-a8f8e953844b" (UID: "3e58b58a-3c6f-4499-a885-a8f8e953844b"). InnerVolumeSpecName "kube-api-access-9wpbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:31:07.559466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.559366 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpbm\" (UniqueName: \"kubernetes.io/projected/3e58b58a-3c6f-4499-a885-a8f8e953844b-kube-api-access-9wpbm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:07.559466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.559401 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:07.559466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:07.559437 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e58b58a-3c6f-4499-a885-a8f8e953844b-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:08.254433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:08.254380 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" Oct 31 03:31:08.257280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:08.257233 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56" event={"ID":"3e58b58a-3c6f-4499-a885-a8f8e953844b","Type":"ContainerDied","Data":"838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83"} Oct 31 03:31:08.257446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:08.257300 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838a9c381555b30650f658ef0832f5df41ba644f361976b2f438807c61247a83" Oct 31 03:31:09.815163 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:09.815119 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:10.577615 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:10.577579 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:14.615967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:14.615923 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:31:15.936557 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:31:15.936518 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:31:20.199489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.199448 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs"] Oct 31 03:31:20.204134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.204099 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lvxs"] Oct 31 03:31:20.224047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.224012 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577"] Oct 31 03:31:20.224438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.224410 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3e58b58a-3c6f-4499-a885-a8f8e953844b" containerName="revoke-keys" Oct 31 03:31:20.224501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.224439 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e58b58a-3c6f-4499-a885-a8f8e953844b" containerName="revoke-keys" Oct 31 03:31:20.224540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.224526 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e58b58a-3c6f-4499-a885-a8f8e953844b" containerName="revoke-keys" Oct 31 03:31:20.227761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.227740 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.230799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.230765 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:31:20.231013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.230776 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:31:20.231013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.230872 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:31:20.231013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.230910 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:31:20.237397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.237368 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577"] Oct 31 03:31:20.257353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.257317 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5091092e-0847-4706-8483-e44c303f93d3" path="/var/lib/kubelet/pods/5091092e-0847-4706-8483-e44c303f93d3/volumes" Oct 31 03:31:20.283067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.283034 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.283252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.283075 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.283252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.283105 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqcs\" (UniqueName: \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.384407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.384363 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.384652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.384460 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.384652 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.384500 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqcs\" (UniqueName: \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.385522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.385497 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.387241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.387215 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.397082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.397052 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqcs\" (UniqueName: \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\") pod \"maas-api-revoke-sub-3d4156f1-4f577\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.544227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.544133 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:20.821724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:20.821695 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577"] Oct 31 03:31:20.823993 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:31:20.823959 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391b0dd9_46c9_41b4_a9dd_cfe7375c335f.slice/crio-ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345 WatchSource:0}: Error finding container ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345: Status 404 returned error can't find the container with id ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345 Oct 31 03:31:21.318353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:21.318316 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" event={"ID":"391b0dd9-46c9-41b4-a9dd-cfe7375c335f","Type":"ContainerStarted","Data":"be60a6672260ac97a514ebf83e13279be5797a67e2d3f2bb95f4f0082c01c3db"} Oct 31 03:31:21.318816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:21.318370 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" event={"ID":"391b0dd9-46c9-41b4-a9dd-cfe7375c335f","Type":"ContainerStarted","Data":"ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345"} Oct 31 03:31:21.335820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:21.335705 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" podStartSLOduration=1.335691009 podStartE2EDuration="1.335691009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:21.334488 +0000 UTC m=+2159.189505141" watchObservedRunningTime="2026-10-31 03:31:21.335691009 +0000 UTC m=+2159.190708140" Oct 31 03:31:22.325398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:22.325360 4556 generic.go:356] "Generic (PLEG): container finished" podID="391b0dd9-46c9-41b4-a9dd-cfe7375c335f" containerID="be60a6672260ac97a514ebf83e13279be5797a67e2d3f2bb95f4f0082c01c3db" exitCode=0 Oct 31 03:31:22.325809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:22.325449 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" event={"ID":"391b0dd9-46c9-41b4-a9dd-cfe7375c335f","Type":"ContainerDied","Data":"be60a6672260ac97a514ebf83e13279be5797a67e2d3f2bb95f4f0082c01c3db"} Oct 31 03:31:23.470971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.470943 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:23.519081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.519045 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\") pod \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " Oct 31 03:31:23.519256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.519160 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\") pod \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " Oct 31 03:31:23.519256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.519204 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\" (UniqueName: \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\") pod \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\" (UID: \"391b0dd9-46c9-41b4-a9dd-cfe7375c335f\") " Oct 31 03:31:23.519517 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.519486 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca" pod "391b0dd9-46c9-41b4-a9dd-cfe7375c335f" (UID: "391b0dd9-46c9-41b4-a9dd-cfe7375c335f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:31:23.521407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.521377 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs" pod "391b0dd9-46c9-41b4-a9dd-cfe7375c335f" (UID: "391b0dd9-46c9-41b4-a9dd-cfe7375c335f"). InnerVolumeSpecName "kube-api-access-cvqcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:31:23.521407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.521398 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca" pod "391b0dd9-46c9-41b4-a9dd-cfe7375c335f" (UID: "391b0dd9-46c9-41b4-a9dd-cfe7375c335f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:31:23.620469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.620332 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:23.620469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.620379 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqcs\" (UniqueName: \"kubernetes.io/projected/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-kube-api-access-cvqcs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:23.620469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:23.620391 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391b0dd9-46c9-41b4-a9dd-cfe7375c335f-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:24.337754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:24.337722 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" Oct 31 03:31:24.337936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:24.337751 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577" event={"ID":"391b0dd9-46c9-41b4-a9dd-cfe7375c335f","Type":"ContainerDied","Data":"ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345"} Oct 31 03:31:24.337936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:24.337803 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7b4ce4282f8ba0eb3ab98370fce308b38b7d971a55594dd46294e853f60345" Oct 31 03:31:26.065254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:26.065210 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:26.195699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:26.195655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:31.048693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:31.048654 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:31:35.062709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:35.062674 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:31:38.641018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:38.640964 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:38.994974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:38.994930 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:31:38.995454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:38.995435 4556 cpu_manager.go:414] "containerMap: removing container" podUID="391b0dd9-46c9-41b4-a9dd-cfe7375c335f" containerName="revoke-keys" Oct 31 03:31:38.995568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:38.995455 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391b0dd9-46c9-41b4-a9dd-cfe7375c335f" containerName="revoke-keys" Oct 31 03:31:38.995627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:38.995575 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="391b0dd9-46c9-41b4-a9dd-cfe7375c335f" containerName="revoke-keys" Oct 31 03:31:39.000230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.000205 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.004060 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.004035 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-01b345-openshift-default-dockercfg-6ddks\"" Oct 31 03:31:39.012798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.012763 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:31:39.079244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.079208 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:39.103833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.103787 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.103833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.103845 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.103906 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.103945 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.103993 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbf4\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.104033 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.104064 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.104100 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.104482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.104139 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205228 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205297 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205330 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205364 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205475 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205517 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205579 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbf4\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205622 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205654 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.205764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205718 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.206048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205927 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.206048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.205984 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.206223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.206199 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.206580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.206560 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.208322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.208297 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.208522 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.208502 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.234363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.234321 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.234545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.234463 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbf4\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\") pod \"e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.318557 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.318466 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:39.393044 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.392968 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:39.603222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.603196 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:31:39.605627 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:31:39.605602 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb971b67_e09a_41e4_b5ad_0c24b8337176.slice/crio-c03243ab9c2e73df18b30de611d4ce2d5e7c9388b430b4cc814af98ca3497b19 WatchSource:0}: Error finding container c03243ab9c2e73df18b30de611d4ce2d5e7c9388b430b4cc814af98ca3497b19: Status 404 returned error can't find the container with id c03243ab9c2e73df18b30de611d4ce2d5e7c9388b430b4cc814af98ca3497b19 Oct 31 03:31:39.607813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.607785 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:39.607892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.607849 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:39.607930 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:39.607890 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:40.441210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:40.441172 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerStarted","Data":"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1"} Oct 31 03:31:40.441659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:40.441232 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerStarted","Data":"c03243ab9c2e73df18b30de611d4ce2d5e7c9388b430b4cc814af98ca3497b19"} Oct 31 03:31:40.463021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:40.462954 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podStartSLOduration=2.462932612 podStartE2EDuration="2.462932612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:40.461727574 +0000 UTC m=+2178.316744736" watchObservedRunningTime="2026-10-31 03:31:40.462932612 +0000 UTC m=+2178.317949754" Oct 31 03:31:41.319146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:41.319100 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:41.324455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:41.324425 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:41.446156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:41.446125 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:41.447309 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:41.447285 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:42.740261 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:42.740230 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:42.874779 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:42.874747 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:43.028971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.028885 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf"] Oct 31 03:31:43.031754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.031719 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4v5kf"] Oct 31 03:31:43.882249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.882212 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:31:43.888330 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.888296 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:43.903607 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.903560 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:31:43.908081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.905382 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:31:43.908081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.905489 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:31:43.910207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.910174 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:31:43.910371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.910359 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:31:43.910461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.910394 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-01b345-serving-cert\"" Oct 31 03:31:43.951698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.951640 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:43.951899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.951712 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqdp\" (UniqueName: \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:43.951899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.951820 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:43.952016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.951912 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:43.996202 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:43.996155 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:44.053234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.053190 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.053491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.053242 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.053491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.053320 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.053491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.053355 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqdp\" (UniqueName: \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.054869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.054834 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.056403 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.056376 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.056701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.056675 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.065871 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.065841 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqdp\" (UniqueName: \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\") pod \"maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.130244 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.130212 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:44.226219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.226183 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:44.256850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.256817 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8869ad49-2264-4ddf-895b-9ca8e5a01b3c" path="/var/lib/kubelet/pods/8869ad49-2264-4ddf-895b-9ca8e5a01b3c/volumes" Oct 31 03:31:44.269479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.269447 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:44.509661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.509553 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:31:44.518994 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:31:44.518952 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eff4e82_a0da_444d_b4de_690836168675.slice/crio-2f7f81919affc21b5573587f3564cde1fb2b54a3c2868fd74cdce939152b83f8 WatchSource:0}: Error finding container 2f7f81919affc21b5573587f3564cde1fb2b54a3c2868fd74cdce939152b83f8: Status 404 returned error can't find the container with id 2f7f81919affc21b5573587f3564cde1fb2b54a3c2868fd74cdce939152b83f8 Oct 31 03:31:44.520605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.520585 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:31:44.817482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:44.817386 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:45.472067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:45.466499 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:45.472067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:45.471408 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" event={"ID":"6eff4e82-a0da-444d-b4de-690836168675","Type":"ContainerStarted","Data":"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303"} Oct 31 03:31:45.472067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:45.471492 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" event={"ID":"6eff4e82-a0da-444d-b4de-690836168675","Type":"ContainerStarted","Data":"2f7f81919affc21b5573587f3564cde1fb2b54a3c2868fd74cdce939152b83f8"} Oct 31 03:31:45.472067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:45.471638 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:45.492954 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:45.492895 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" podStartSLOduration=1.959590662 podStartE2EDuration="2.492878593s" totalImagesPullingTime="533.287931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:45.490784981 +0000 UTC m=+2183.345802123" watchObservedRunningTime="2026-10-31 03:31:45.492878593 +0000 UTC m=+2183.347895797" Oct 31 03:31:45.938059 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:31:45.938021 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:31:46.157148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:46.157106 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:31:46.276845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:46.276753 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:46.987082 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:46.987027 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:47.563702 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:47.563649 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:48.192434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:48.192367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:48.761731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:48.761689 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.458900 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:49.458863 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:50.116731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:50.116697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:50.655590 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:50.655548 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:51.311775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:51.311736 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:51.497752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:51.497711 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:31:51.950964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:51.950931 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:52.634061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:52.634006 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:53.255337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:53.255298 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:53.860530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:53.860474 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:54.462509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:54.462467 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:55.066353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.066321 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:55.541128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.541045 4556 generic.go:356] "Generic (PLEG): container finished" podID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerID="8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1" exitCode=0 Oct 31 03:31:55.541128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.541117 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerDied","Data":"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1"} Oct 31 03:31:55.541661 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.541538 4556 scope.go:122] "RemoveContainer" containerID="8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1" Oct 31 03:31:55.542723 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.542697 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:55.542786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.542762 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:55.542835 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.542799 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:31:55.847074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:55.847034 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:56.446624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:56.446589 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:56.534200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:56.534168 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:56.548717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:56.548681 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerStarted","Data":"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791"} Oct 31 03:31:57.062709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.062661 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:57.319002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.318902 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:31:57.320182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.320153 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 03:31:57.320318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.320214 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 03:31:57.739118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.739072 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:57.999214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:57.997359 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.319554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.319457 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 03:31:58.319554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.319518 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 03:31:58.345624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.345581 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.644750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.644655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.845404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.845363 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:58.992446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:58.991021 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:59.132686 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.132655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:59.319154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.319069 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 03:31:59.319154 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.319123 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 03:31:59.443461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.442453 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:31:59.447775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.447714 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.454475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.453468 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-01b345\"/\"simulator-single-main-01b345-kserve-self-signed-certs\"" Oct 31 03:31:59.454832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.454655 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-01b345\"/\"default-dockercfg-nf5m4\"" Oct 31 03:31:59.457302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.457274 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:31:59.513512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.513475 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-01b345\"/\"kube-root-ca.crt\"" Oct 31 03:31:59.513696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.513528 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-01b345\"/\"openshift-service-ca.crt\"" Oct 31 03:31:59.526667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.526619 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:59.540909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.540877 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.541049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.540931 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.541091 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.541071 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.541128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.541103 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.541193 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.541174 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6kh\" (UniqueName: \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.541254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.541243 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.642804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.642704 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.642804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.642762 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6kh\" (UniqueName: \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.642824 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.642874 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.642921 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643073 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.643005 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.643092 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.643312 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.643355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.643350 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.645576 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.645547 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.645814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.645795 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.654613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.654586 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6kh\" (UniqueName: \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\") pod \"simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.765910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.765877 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:31:59.787775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:31:59.787013 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:00.095213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.095157 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:32:00.319072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.318981 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 03:32:00.319072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.319038 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 03:32:00.365185 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.365136 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:00.594197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.593367 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" event={"ID":"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5","Type":"ContainerStarted","Data":"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051"} Oct 31 03:32:00.594197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.593444 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" event={"ID":"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5","Type":"ContainerStarted","Data":"a1e56e882248d1c1505a623e582d7f8198bf442dd9cb5864150e1ece8631003c"} Oct 31 03:32:00.616061 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.615988 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" podStartSLOduration=1.615971244 podStartE2EDuration="1.615971244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:00.613173227 +0000 UTC m=+2198.468190362" watchObservedRunningTime="2026-10-31 03:32:00.615971244 +0000 UTC m=+2198.470988426" Oct 31 03:32:00.766341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:00.766306 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:32:01.226286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:01.226253 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:32:01.319883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:01.319847 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 31 03:32:01.320081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:01.319904 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 31 03:32:02.172045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.171997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:02.306593 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.306554 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:02.323956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.323923 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:32:02.324371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.324350 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:32:02.325144 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.325125 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:32:02.760287 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:02.760247 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:03.360986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:03.360951 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:04.084253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:04.084155 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:04.741205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:04.741165 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:05.382697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:05.382662 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:05.993708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:05.993670 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:06.548081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:06.548048 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:07.161981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:07.161933 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:07.814209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:07.814171 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:08.458477 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:08.458437 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.080325 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:09.080273 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.695406 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:09.695366 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.766442 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:09.766380 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:32:09.779501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:09.779469 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:32:09.854211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:09.854177 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:10.378707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:10.378674 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:10.657382 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:10.657355 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:32:10.679017 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:10.678984 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:10.980976 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:10.980877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.635464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:11.635401 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.356190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:12.356152 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.974935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:12.974890 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:13.680374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:13.680336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:14.281655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:14.281625 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:14.954319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:14.954280 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:15.583501 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:15.583468 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:15.936443 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:32:15.936387 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:32:16.270970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:16.270855 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:17.065651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:17.065616 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:17.713264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:17.713230 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:17.951436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:17.951388 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:32:18.353338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:18.353279 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:19.154153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:19.154105 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:19.753919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:19.753877 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:20.452245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:20.452203 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.114989 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:21.114951 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.761641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:21.761604 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:22.467189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:22.467146 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:23.102443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:23.102396 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:23.672478 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:23.672435 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:24.267107 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:24.267047 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:25.071395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:25.071362 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:25.667804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:25.667760 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:26.263074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:26.263044 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:26.966009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:26.965969 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:27.687482 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:27.687438 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:28.363747 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:28.363651 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:28.720856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:28.720815 4556 scope.go:122] "RemoveContainer" containerID="26745c117af09a92812f1d08c4cb59d59252fc9039b81495df002b166159fadf" Oct 31 03:32:28.730290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:28.730262 4556 scope.go:122] "RemoveContainer" containerID="61f89b928cfc78421cd979bc55c0060e03151cb6f8e9abf46d384d8b5fc712ad" Oct 31 03:32:29.076190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:29.076106 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:29.742961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:29.742925 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:30.355927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:30.355880 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:31.078764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:31.078733 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:31.666905 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:31.666869 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.271753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:32.271711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.944483 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:32.944444 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:33.557089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:33.557055 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.173472 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:34.173404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.785949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:34.785898 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:35.457998 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:35.457965 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:36.025713 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:36.025677 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn"] Oct 31 03:32:36.030253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:36.030223 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mq2sn"] Oct 31 03:32:36.089387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:36.089346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:36.256129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:36.256093 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dba02ef-edbf-4e88-ad78-c2469a46de7b" path="/var/lib/kubelet/pods/5dba02ef-edbf-4e88-ad78-c2469a46de7b/volumes" Oct 31 03:32:36.679120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:36.679083 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:37.339885 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:37.339849 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:37.969984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:37.969941 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:38.593657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:38.593619 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:39.302486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:39.302443 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:39.883000 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:39.882966 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:40.462629 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:40.462591 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:41.173407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:41.173368 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:41.390545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:41.390514 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:42.018845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:42.018785 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:42.149069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:42.149034 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:32:42.717454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:42.717383 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:43.385660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:43.385618 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:44.156219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:44.156174 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:44.762263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:44.762216 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:45.456712 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:45.456661 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:45.936615 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:32:45.936572 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:32:46.077461 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:46.077388 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.312127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:46.312040 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:32:46.424910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:46.424878 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.775190 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:46.775142 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:47.558226 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:47.558187 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:48.155643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:48.155598 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:48.859142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:48.859094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:49.458545 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:49.458495 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:50.066232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:50.066196 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:50.654290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:50.654258 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:51.261813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:51.261748 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:51.716301 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:51.716261 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:51.916917 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:51.916873 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.043883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.043790 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.425731 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.425693 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:32:52.425984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.425951 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" podUID="6eff4e82-a0da-444d-b4de-690836168675" containerName="maas-api" containerID="cri-o://c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303" gracePeriod=30 Oct 31 03:32:52.520693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.520655 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.659691 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.659599 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.711614 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.711588 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:32:52.785130 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.785096 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.840174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.840124 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\") pod \"6eff4e82-a0da-444d-b4de-690836168675\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " Oct 31 03:32:52.840459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.840197 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\") pod \"6eff4e82-a0da-444d-b4de-690836168675\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " Oct 31 03:32:52.840459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.840250 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\" (UniqueName: \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\") pod \"6eff4e82-a0da-444d-b4de-690836168675\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " Oct 31 03:32:52.840641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.840531 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\") pod \"6eff4e82-a0da-444d-b4de-690836168675\" (UID: \"6eff4e82-a0da-444d-b4de-690836168675\") " Oct 31 03:32:52.841680 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.841644 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca" pod "6eff4e82-a0da-444d-b4de-690836168675" (UID: "6eff4e82-a0da-444d-b4de-690836168675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:52.843898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.843856 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls" pod "6eff4e82-a0da-444d-b4de-690836168675" (UID: "6eff4e82-a0da-444d-b4de-690836168675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:52.844025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.843898 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs" pod "6eff4e82-a0da-444d-b4de-690836168675" (UID: "6eff4e82-a0da-444d-b4de-690836168675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:52.844025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.843921 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp" pod "6eff4e82-a0da-444d-b4de-690836168675" (UID: "6eff4e82-a0da-444d-b4de-690836168675"). InnerVolumeSpecName "kube-api-access-cjqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:52.884266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.884230 4556 generic.go:356] "Generic (PLEG): container finished" podID="6eff4e82-a0da-444d-b4de-690836168675" containerID="c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303" exitCode=0 Oct 31 03:32:52.884470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.884299 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" Oct 31 03:32:52.884470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.884316 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" event={"ID":"6eff4e82-a0da-444d-b4de-690836168675","Type":"ContainerDied","Data":"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303"} Oct 31 03:32:52.884470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.884393 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg" event={"ID":"6eff4e82-a0da-444d-b4de-690836168675","Type":"ContainerDied","Data":"2f7f81919affc21b5573587f3564cde1fb2b54a3c2868fd74cdce939152b83f8"} Oct 31 03:32:52.884470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.884442 4556 scope.go:122] "RemoveContainer" containerID="c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303" Oct 31 03:32:52.894798 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.894778 4556 scope.go:122] "RemoveContainer" containerID="c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303" Oct 31 03:32:52.895101 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:32:52.895078 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303\": container with ID starting with c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303 not found: ID does not exist" containerID="c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303" Oct 31 03:32:52.895171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.895115 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303"} err="failed to get container status \"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303\": rpc error: code = NotFound desc = could not find container \"c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303\": container with ID starting with c298e1e0fcd95e550767724ad6ab6a40c25c3708b419131a02f5f1e7a316d303 not found: ID does not exist" Oct 31 03:32:52.905498 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.905461 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:32:52.909480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.909448 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-01b345-694f8899f7-496kg"] Oct 31 03:32:52.942839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.942749 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.942839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.942783 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eff4e82-a0da-444d-b4de-690836168675-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.942839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.942795 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eff4e82-a0da-444d-b4de-690836168675-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.942839 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:52.942805 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjqdp\" (UniqueName: \"kubernetes.io/projected/6eff4e82-a0da-444d-b4de-690836168675-kube-api-access-cjqdp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:53.346574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:53.346489 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:54.141119 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:54.141083 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:32:54.256667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:54.256624 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eff4e82-a0da-444d-b4de-690836168675" path="/var/lib/kubelet/pods/6eff4e82-a0da-444d-b4de-690836168675/volumes" Oct 31 03:32:56.027690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:56.027645 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb"] Oct 31 03:32:56.030638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:56.030608 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-cfklb"] Oct 31 03:32:56.256697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:56.256666 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09945448-737d-4899-9e10-d391fb60fd7b" path="/var/lib/kubelet/pods/09945448-737d-4899-9e10-d391fb60fd7b/volumes" Oct 31 03:32:57.795966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:57.795898 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:59.823524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:59.823481 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:59.966634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:32:59.966597 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:00.146400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:00.146311 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:33:00.146677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:00.146625 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" containerID="cri-o://290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791" gracePeriod=30 Oct 31 03:33:00.222101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:00.222064 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:01.333046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.333007 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" podUID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" containerName="main" containerID="cri-o://be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051" gracePeriod=45 Oct 31 03:33:01.492983 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.492937 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577"] Oct 31 03:33:01.497724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.497694 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4f577"] Oct 31 03:33:01.519587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.519536 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn"] Oct 31 03:33:01.520215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.520193 4556 cpu_manager.go:414] "containerMap: removing container" podUID="6eff4e82-a0da-444d-b4de-690836168675" containerName="maas-api" Oct 31 03:33:01.520328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.520215 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eff4e82-a0da-444d-b4de-690836168675" containerName="maas-api" Oct 31 03:33:01.520328 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.520313 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eff4e82-a0da-444d-b4de-690836168675" containerName="maas-api" Oct 31 03:33:01.524034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.524009 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.527462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.527322 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhm8p\" (UniqueName: \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.527637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.527480 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.527637 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.527609 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.530087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.530062 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:33:01.532814 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.532706 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:33:01.532984 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.532822 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:33:01.534761 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.534626 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:33:01.536436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.536396 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn"] Oct 31 03:33:01.598894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.598823 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:33:01.629005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.628958 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629034 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629086 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629162 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629266 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\" (UniqueName: \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629310 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\") pod \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\" (UID: \"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5\") " Oct 31 03:33:01.629460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629308 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:01.629669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629531 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:01.629669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629621 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhm8p\" (UniqueName: \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.629790 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.629690 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.631505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.630136 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.631505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.630521 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.631505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.630546 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.631505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.630624 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:01.631505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.631454 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.632874 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.632834 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.633004 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.632964 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "kube-api-access-lf6kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:01.633869 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.633839 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:01.634125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.634101 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs" pod "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" (UID: "3776dccb-e3f1-42a1-afc1-92c42f7fa3d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:33:01.644775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.644732 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhm8p\" (UniqueName: \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\") pod \"maas-api-revoke-sub-3d4156f1-5w5qn\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.731624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.731585 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.731624 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.731622 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.731865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.731635 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf6kh\" (UniqueName: \"kubernetes.io/projected/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-kube-api-access-lf6kh\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.731865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.731647 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:01.844833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.844796 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:01.943206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.943165 4556 generic.go:356] "Generic (PLEG): container finished" podID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" containerID="be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051" exitCode=0 Oct 31 03:33:01.943455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.943216 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" Oct 31 03:33:01.943455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.943265 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" event={"ID":"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5","Type":"ContainerDied","Data":"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051"} Oct 31 03:33:01.943455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.943323 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr" event={"ID":"3776dccb-e3f1-42a1-afc1-92c42f7fa3d5","Type":"ContainerDied","Data":"a1e56e882248d1c1505a623e582d7f8198bf442dd9cb5864150e1ece8631003c"} Oct 31 03:33:01.943455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.943343 4556 scope.go:122] "RemoveContainer" containerID="be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051" Oct 31 03:33:01.957955 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.957928 4556 scope.go:122] "RemoveContainer" containerID="be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051" Oct 31 03:33:01.958381 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:33:01.958352 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051\": container with ID starting with be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051 not found: ID does not exist" containerID="be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051" Oct 31 03:33:01.958512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.958392 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051"} err="failed to get container status \"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051\": rpc error: code = NotFound desc = could not find container \"be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051\": container with ID starting with be502cb846d242b069214ddc754e3ef69fde96f3c05aea465b3a37d203d9f051 not found: ID does not exist" Oct 31 03:33:01.970681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.970641 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:33:01.974084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:01.974052 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-01b345/simulator-single-main-01b345-kserve-6fbcb98bfc-nthkr"] Oct 31 03:33:02.122353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.122072 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn"] Oct 31 03:33:02.255861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.255817 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" path="/var/lib/kubelet/pods/3776dccb-e3f1-42a1-afc1-92c42f7fa3d5/volumes" Oct 31 03:33:02.256333 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.256312 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391b0dd9-46c9-41b4-a9dd-cfe7375c335f" path="/var/lib/kubelet/pods/391b0dd9-46c9-41b4-a9dd-cfe7375c335f/volumes" Oct 31 03:33:02.326788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.326742 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:33:02.326949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.326803 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:33:02.950290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.950252 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" event={"ID":"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3","Type":"ContainerStarted","Data":"ecf5937e9054c52edcad93ab4f51f52ede0f113f93c29560a30df1866180fad2"} Oct 31 03:33:02.950769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.950319 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" event={"ID":"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3","Type":"ContainerStarted","Data":"6734acf37f910b2a1f12de0032efad6aaee070e84e5fff1c885f281d819614b6"} Oct 31 03:33:02.969660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:02.969606 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" podStartSLOduration=1.969585925 podStartE2EDuration="1.969585925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:02.967498728 +0000 UTC m=+2260.822515866" watchObservedRunningTime="2026-10-31 03:33:02.969585925 +0000 UTC m=+2260.824603074" Oct 31 03:33:03.958730 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:03.958696 4556 generic.go:356] "Generic (PLEG): container finished" podID="ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" containerID="ecf5937e9054c52edcad93ab4f51f52ede0f113f93c29560a30df1866180fad2" exitCode=0 Oct 31 03:33:03.959146 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:03.958792 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" event={"ID":"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3","Type":"ContainerDied","Data":"ecf5937e9054c52edcad93ab4f51f52ede0f113f93c29560a30df1866180fad2"} Oct 31 03:33:05.106212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.106182 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:05.168142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.168109 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\") pod \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " Oct 31 03:33:05.168299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.168205 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\" (UniqueName: \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\") pod \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " Oct 31 03:33:05.168299 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.168242 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\") pod \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\" (UID: \"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3\") " Oct 31 03:33:05.168531 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.168509 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca" pod "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" (UID: "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:05.168709 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.168688 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.170338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.170306 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca" pod "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" (UID: "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:33:05.170462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.170311 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p" pod "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" (UID: "ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3"). InnerVolumeSpecName "kube-api-access-mhm8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:05.269626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.269594 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhm8p\" (UniqueName: \"kubernetes.io/projected/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-kube-api-access-mhm8p\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.269626 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.269625 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.387462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.387403 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:33:05.470694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470659 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.470916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470723 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.470916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470762 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.470916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470788 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.470916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470855 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.470916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470902 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.471219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.470939 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.471219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471000 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.471219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471048 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\") pod \"fb971b67-e09a-41e4-b5ad-0c24b8337176\" (UID: \"fb971b67-e09a-41e4-b5ad-0c24b8337176\") " Oct 31 03:33:05.471219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471127 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:05.471505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471237 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:05.471505 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471450 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:05.471621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471548 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.471621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471565 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.471621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471573 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:05.471793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.471772 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:05.473345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.473318 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:33:05.473677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.473654 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:05.473677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.473666 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:05.473805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.473761 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4" pod "fb971b67-e09a-41e4-b5ad-0c24b8337176" (UID: "fb971b67-e09a-41e4-b5ad-0c24b8337176"). InnerVolumeSpecName "kube-api-access-qhbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572257 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572289 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572299 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhbf4\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-kube-api-access-qhbf4\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572309 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572318 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb971b67-e09a-41e4-b5ad-0c24b8337176-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572326 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb971b67-e09a-41e4-b5ad-0c24b8337176-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.572350 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.572337 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb971b67-e09a-41e4-b5ad-0c24b8337176-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:05.970733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.970698 4556 generic.go:356] "Generic (PLEG): container finished" podID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerID="290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791" exitCode=0 Oct 31 03:33:05.970912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.970764 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" Oct 31 03:33:05.970912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.970778 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerDied","Data":"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791"} Oct 31 03:33:05.970912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.970831 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25" event={"ID":"fb971b67-e09a-41e4-b5ad-0c24b8337176","Type":"ContainerDied","Data":"c03243ab9c2e73df18b30de611d4ce2d5e7c9388b430b4cc814af98ca3497b19"} Oct 31 03:33:05.970912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.970862 4556 scope.go:122] "RemoveContainer" containerID="290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791" Oct 31 03:33:05.972487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.972457 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" event={"ID":"ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3","Type":"ContainerDied","Data":"6734acf37f910b2a1f12de0032efad6aaee070e84e5fff1c885f281d819614b6"} Oct 31 03:33:05.972636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.972519 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6734acf37f910b2a1f12de0032efad6aaee070e84e5fff1c885f281d819614b6" Oct 31 03:33:05.972636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.972473 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn" Oct 31 03:33:05.980456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.980399 4556 scope.go:122] "RemoveContainer" containerID="8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1" Oct 31 03:33:05.989683 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.989661 4556 scope.go:122] "RemoveContainer" containerID="290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791" Oct 31 03:33:05.989990 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:33:05.989967 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791\": container with ID starting with 290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791 not found: ID does not exist" containerID="290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791" Oct 31 03:33:05.990065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.990002 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791"} err="failed to get container status \"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791\": rpc error: code = NotFound desc = could not find container \"290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791\": container with ID starting with 290c6158b3686b90307e3648fd9558e1d1511610ab1cef85ceb2784fe7da2791 not found: ID does not exist" Oct 31 03:33:05.990065 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.990026 4556 scope.go:122] "RemoveContainer" containerID="8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1" Oct 31 03:33:05.990276 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:33:05.990258 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1\": container with ID starting with 8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1 not found: ID does not exist" containerID="8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1" Oct 31 03:33:05.990314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:05.990283 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1"} err="failed to get container status \"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1\": rpc error: code = NotFound desc = could not find container \"8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1\": container with ID starting with 8e1e64509ae892999425d342e1120abc087ab84f27c2fb4e1bf307b4ff7175f1 not found: ID does not exist" Oct 31 03:33:06.001057 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:06.001027 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:33:06.007034 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:06.007002 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-01b345-openshift-default-64dd99647cdml25"] Oct 31 03:33:06.256441 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:06.256345 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" path="/var/lib/kubelet/pods/fb971b67-e09a-41e4-b5ad-0c24b8337176/volumes" Oct 31 03:33:07.025451 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:07.025377 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8"] Oct 31 03:33:07.027799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:07.027762 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-plfq8"] Oct 31 03:33:08.257018 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:08.256976 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c382ec-1eb7-4f74-ad97-7c75a0d7c763" path="/var/lib/kubelet/pods/c2c382ec-1eb7-4f74-ad97-7c75a0d7c763/volumes" Oct 31 03:33:09.926566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:09.926534 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:10.671708 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:10.671664 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:12.354526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:12.354481 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:33:15.936394 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:33:15.936354 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:33:16.447558 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:16.447520 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:33:23.589519 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:23.589404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:24.088310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:24.088253 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:28.756850 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:28.756811 4556 scope.go:122] "RemoveContainer" containerID="c7340a4100204ddbe37ea955a43f85fc87fe8aed9c5de3fce129c00eafac13a3" Oct 31 03:33:30.378704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:30.378663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:32.425967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:32.425927 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:33.025815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:33.025781 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5"] Oct 31 03:33:33.028300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:33.028273 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tw2m5"] Oct 31 03:33:34.256673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:34.256581 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fb0b3c-bd36-4dff-a139-28868e5eff8b" path="/var/lib/kubelet/pods/c5fb0b3c-bd36-4dff-a139-28868e5eff8b/volumes" Oct 31 03:33:43.024566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:43.024528 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm"] Oct 31 03:33:43.026977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:43.026951 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-lxvqm"] Oct 31 03:33:44.256449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:33:44.256396 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c3f3e5-105e-44d3-b106-4a9613b22a16" path="/var/lib/kubelet/pods/43c3f3e5-105e-44d3-b106-4a9613b22a16/volumes" Oct 31 03:33:45.936952 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:33:45.936911 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:34:04.026011 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:04.025968 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8"] Oct 31 03:34:04.029272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:04.029243 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4j8f8"] Oct 31 03:34:04.256575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:04.256535 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b5d86e-731e-4c17-a1cb-ef23a49da733" path="/var/lib/kubelet/pods/c2b5d86e-731e-4c17-a1cb-ef23a49da733/volumes" Oct 31 03:34:10.024354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:10.024308 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:10.733064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:10.733034 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:15.937033 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:34:15.936990 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:34:22.033495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:22.033449 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m"] Oct 31 03:34:22.038266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:22.038231 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4227m"] Oct 31 03:34:22.256865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:22.256825 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54323503-6e68-4dbe-9874-622c2fe4afe6" path="/var/lib/kubelet/pods/54323503-6e68-4dbe-9874-622c2fe4afe6/volumes" Oct 31 03:34:28.794426 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:28.794391 4556 scope.go:122] "RemoveContainer" containerID="1638f86ca21f4178a4efa9cc8e554d5d26a6ce010ee676c781e379baace8db0f" Oct 31 03:34:28.803085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:28.803058 4556 scope.go:122] "RemoveContainer" containerID="ee07a1ea5ef6018fe25c20544dd1ffc3c638f72f789b2bf7899ba714d1eb0b76" Oct 31 03:34:28.811493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:28.811404 4556 scope.go:122] "RemoveContainer" containerID="880ff203b9b8b8339b4be2a3654d56d31916577a1431c6707449ad97064069c5" Oct 31 03:34:41.024633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:41.024586 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn"] Oct 31 03:34:41.027291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:41.027258 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-896gn"] Oct 31 03:34:42.133466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:42.133429 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:34:42.258001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:42.257962 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e" path="/var/lib/kubelet/pods/67b45e4e-5e1f-4539-a6fb-bdb4b4bda96e/volumes" Oct 31 03:34:45.935887 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:34:45.935842 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:34:46.244966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.244885 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n"] Oct 31 03:34:46.245292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245279 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" containerName="main" Oct 31 03:34:46.245335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245294 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" containerName="main" Oct 31 03:34:46.245335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245324 4556 cpu_manager.go:414] "containerMap: removing container" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245329 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245342 4556 cpu_manager.go:414] "containerMap: removing container" podUID="ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" containerName="revoke-keys" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245347 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" containerName="revoke-keys" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245355 4556 cpu_manager.go:414] "containerMap: removing container" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245359 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245443 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245456 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" containerName="revoke-keys" Oct 31 03:34:46.245467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245464 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb971b67-e09a-41e4-b5ad-0c24b8337176" containerName="istio-proxy" Oct 31 03:34:46.245696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.245471 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3776dccb-e3f1-42a1-afc1-92c42f7fa3d5" containerName="main" Oct 31 03:34:46.250227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.250200 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.256345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.256315 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:34:46.256613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.256402 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:34:46.256613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.256477 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:34:46.256613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.256502 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:34:46.259090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.259068 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n"] Oct 31 03:34:46.348912 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.348866 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.349101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.348923 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mws9m\" (UniqueName: \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.349101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.349075 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.450118 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.450068 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.450343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.450166 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.450343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.450203 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws9m\" (UniqueName: \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.450880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.450859 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.452579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.452559 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.462314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.462292 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mws9m\" (UniqueName: \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\") pod \"maas-api-revoke-sub-d576e68b-dfc9n\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.565962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.565875 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:46.841491 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:46.841463 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n"] Oct 31 03:34:46.844168 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:34:46.844137 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2792d0_25a2_4088_b902_b0bd58b6f832.slice/crio-fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2 WatchSource:0}: Error finding container fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2: Status 404 returned error can't find the container with id fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2 Oct 31 03:34:47.554647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:47.554552 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" event={"ID":"9e2792d0-25a2-4088-b902-b0bd58b6f832","Type":"ContainerStarted","Data":"70673fad6419c437047ab41d4d3d267d9ef13b89f7b5f58a15ee3801e03d4864"} Oct 31 03:34:47.554647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:47.554612 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" event={"ID":"9e2792d0-25a2-4088-b902-b0bd58b6f832","Type":"ContainerStarted","Data":"fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2"} Oct 31 03:34:47.572354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:47.572292 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" podStartSLOduration=1.572273013 podStartE2EDuration="1.572273013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:47.570860348 +0000 UTC m=+2365.425877488" watchObservedRunningTime="2026-10-31 03:34:47.572273013 +0000 UTC m=+2365.427290157" Oct 31 03:34:48.561359 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:48.561317 4556 generic.go:356] "Generic (PLEG): container finished" podID="9e2792d0-25a2-4088-b902-b0bd58b6f832" containerID="70673fad6419c437047ab41d4d3d267d9ef13b89f7b5f58a15ee3801e03d4864" exitCode=0 Oct 31 03:34:48.561845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:48.561358 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" event={"ID":"9e2792d0-25a2-4088-b902-b0bd58b6f832","Type":"ContainerDied","Data":"70673fad6419c437047ab41d4d3d267d9ef13b89f7b5f58a15ee3801e03d4864"} Oct 31 03:34:49.700599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.700571 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:49.780070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.780034 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\" (UniqueName: \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\") pod \"9e2792d0-25a2-4088-b902-b0bd58b6f832\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " Oct 31 03:34:49.780274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.780141 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\") pod \"9e2792d0-25a2-4088-b902-b0bd58b6f832\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " Oct 31 03:34:49.780274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.780189 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\") pod \"9e2792d0-25a2-4088-b902-b0bd58b6f832\" (UID: \"9e2792d0-25a2-4088-b902-b0bd58b6f832\") " Oct 31 03:34:49.780572 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.780546 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca" pod "9e2792d0-25a2-4088-b902-b0bd58b6f832" (UID: "9e2792d0-25a2-4088-b902-b0bd58b6f832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:34:49.782212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.782186 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca" pod "9e2792d0-25a2-4088-b902-b0bd58b6f832" (UID: "9e2792d0-25a2-4088-b902-b0bd58b6f832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:34:49.782307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.782231 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m" pod "9e2792d0-25a2-4088-b902-b0bd58b6f832" (UID: "9e2792d0-25a2-4088-b902-b0bd58b6f832"). InnerVolumeSpecName "kube-api-access-mws9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:49.881224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.881132 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:49.881224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.881168 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2792d0-25a2-4088-b902-b0bd58b6f832-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:49.881224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:49.881178 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mws9m\" (UniqueName: \"kubernetes.io/projected/9e2792d0-25a2-4088-b902-b0bd58b6f832-kube-api-access-mws9m\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:50.571963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:50.571871 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" Oct 31 03:34:50.571963 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:50.571914 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n" event={"ID":"9e2792d0-25a2-4088-b902-b0bd58b6f832","Type":"ContainerDied","Data":"fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2"} Oct 31 03:34:50.572207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:50.571971 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe58268c23c7b67016f1f9c01f083f59d2357be0748842f7c75501592cdef5e2" Oct 31 03:34:57.056143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:57.056102 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:34:59.025956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:59.025915 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz"] Oct 31 03:34:59.028468 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:34:59.028442 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nqxnz"] Oct 31 03:35:00.257083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:00.257046 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8e35c1-64ae-4232-a740-fc86722741ba" path="/var/lib/kubelet/pods/6c8e35c1-64ae-4232-a740-fc86722741ba/volumes" Oct 31 03:35:00.865587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:00.865543 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:35:05.191563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.191474 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd"] Oct 31 03:35:05.191969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.191890 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9e2792d0-25a2-4088-b902-b0bd58b6f832" containerName="revoke-keys" Oct 31 03:35:05.191969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.191902 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2792d0-25a2-4088-b902-b0bd58b6f832" containerName="revoke-keys" Oct 31 03:35:05.191969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.191956 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2792d0-25a2-4088-b902-b0bd58b6f832" containerName="revoke-keys" Oct 31 03:35:05.195290 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.195270 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.198265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.198237 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:05.198449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.198339 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:35:05.198535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.198447 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:05.198595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.198528 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:35:05.207207 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.207179 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd"] Oct 31 03:35:05.286263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.286222 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452jn\" (UniqueName: \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.286489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.286282 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.286489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.286367 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.386971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.386927 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.386971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.386973 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.387780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.387751 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.387941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.387786 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452jn\" (UniqueName: \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.389666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.389642 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.400079 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.400048 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-452jn\" (UniqueName: \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\") pod \"maas-api-revoke-sub-d2bc93bc-6pmpd\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.508628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.508531 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:05.784083 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:05.783993 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd"] Oct 31 03:35:05.787161 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:35:05.787128 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cd67c2_46ef_4a9e_920d_f41bbe7e9898.slice/crio-2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66 WatchSource:0}: Error finding container 2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66: Status 404 returned error can't find the container with id 2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66 Oct 31 03:35:06.675752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:06.675718 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" event={"ID":"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898","Type":"ContainerStarted","Data":"42a078052608cab3ce57f81ce50e754e89efc8d08034271c671d6eb0fcf775de"} Oct 31 03:35:06.676160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:06.675782 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" event={"ID":"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898","Type":"ContainerStarted","Data":"2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66"} Oct 31 03:35:06.696351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:06.696289 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" podStartSLOduration=1.6962739820000001 podStartE2EDuration="1.696273982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:06.694270179 +0000 UTC m=+2384.549287319" watchObservedRunningTime="2026-10-31 03:35:06.696273982 +0000 UTC m=+2384.551291117" Oct 31 03:35:07.682153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:07.682117 4556 generic.go:356] "Generic (PLEG): container finished" podID="e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" containerID="42a078052608cab3ce57f81ce50e754e89efc8d08034271c671d6eb0fcf775de" exitCode=0 Oct 31 03:35:07.682153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:07.682158 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" event={"ID":"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898","Type":"ContainerDied","Data":"42a078052608cab3ce57f81ce50e754e89efc8d08034271c671d6eb0fcf775de"} Oct 31 03:35:08.824909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.824878 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:08.924953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.924921 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\" (UniqueName: \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\") pod \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " Oct 31 03:35:08.925126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.924999 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\") pod \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " Oct 31 03:35:08.925197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.925179 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\") pod \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\" (UID: \"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898\") " Oct 31 03:35:08.925530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.925504 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca" pod "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" (UID: "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:08.925675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.925612 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:08.927496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.927460 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca" pod "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" (UID: "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:08.927608 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:08.927471 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn" pod "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" (UID: "e7cd67c2-46ef-4a9e-920d-f41bbe7e9898"). InnerVolumeSpecName "kube-api-access-452jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:09.027195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:09.027104 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452jn\" (UniqueName: \"kubernetes.io/projected/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-kube-api-access-452jn\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:09.027195 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:09.027141 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:09.694704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:09.694664 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" event={"ID":"e7cd67c2-46ef-4a9e-920d-f41bbe7e9898","Type":"ContainerDied","Data":"2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66"} Oct 31 03:35:09.694704 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:09.694691 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd" Oct 31 03:35:09.694943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:09.694713 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c449bec3c85514502796582cab109567f94a3b88c9bc82d7980b14236d9cd66" Oct 31 03:35:10.009595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:10.009502 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:10.762262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:10.762230 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:15.937060 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:35:15.937014 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:35:16.222535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:16.222447 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:35:16.831980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:16.831909 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:17.139600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.139503 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:35:17.140037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.139951 4556 cpu_manager.go:414] "containerMap: removing container" podUID="e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" containerName="revoke-keys" Oct 31 03:35:17.140037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.139969 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" containerName="revoke-keys" Oct 31 03:35:17.140037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.140029 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" containerName="revoke-keys" Oct 31 03:35:17.143479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.143452 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.146677 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.146637 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-31e475-openshift-default-dockercfg-zhh8w\"" Oct 31 03:35:17.158042 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.158007 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:35:17.212530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212489 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212530 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212535 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212669 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xxf\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212786 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212834 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212856 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212888 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.212951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212941 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.213124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.212989 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.255298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.255256 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:17.314224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314175 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314224 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314230 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314258 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314293 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314324 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314344 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314372 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314401 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314493 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xxf\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314705 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314861 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.314899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.314865 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.315215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.315195 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.315310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.315291 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.316854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.316833 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.316933 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.316889 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.326529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.326495 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xxf\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.326701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.326578 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\") pod \"e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.459744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.459697 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:17.597263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.597232 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:17.749293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.749201 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:35:17.752408 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:35:17.752378 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54313f3f_f10e_4281_ae4b_c71f0112aec2.slice/crio-e16b6ad0ced26865ff7ddd14ca5730777cefb3b1c963b494c5c77b344b3acc61 WatchSource:0}: Error finding container e16b6ad0ced26865ff7ddd14ca5730777cefb3b1c963b494c5c77b344b3acc61: Status 404 returned error can't find the container with id e16b6ad0ced26865ff7ddd14ca5730777cefb3b1c963b494c5c77b344b3acc61 Oct 31 03:35:17.754724 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.754692 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:17.754830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.754759 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:17.754830 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:17.754791 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:18.740740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:18.740706 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerStarted","Data":"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87"} Oct 31 03:35:18.741270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:18.741234 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerStarted","Data":"e16b6ad0ced26865ff7ddd14ca5730777cefb3b1c963b494c5c77b344b3acc61"} Oct 31 03:35:18.766367 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:18.766304 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podStartSLOduration=1.766287698 podStartE2EDuration="1.766287698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:18.764475928 +0000 UTC m=+2396.619493071" watchObservedRunningTime="2026-10-31 03:35:18.766287698 +0000 UTC m=+2396.621304840" Oct 31 03:35:19.460298 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:19.460254 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:19.465701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:19.465671 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:19.746322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:19.746237 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:19.747351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:19.747332 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:21.215532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:21.215493 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:21.411030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:21.410988 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:22.265289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.265254 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:35:22.269008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.268976 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.283994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.283961 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:35:22.287748 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.287717 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:22.287945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.287820 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:22.319386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.319352 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-31e475-serving-cert\"" Oct 31 03:35:22.319627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.319473 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:35:22.319627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.319533 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:35:22.364547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.364496 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.364760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.364643 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.364760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.364696 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.364867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.364841 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj94l\" (UniqueName: \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.458513 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.458480 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:22.465894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.465860 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj94l\" (UniqueName: \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.466103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.466004 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.466103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.466060 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.466242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.466174 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.468131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.468084 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:35:22.468305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.468137 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-31e475-serving-cert\"" Oct 31 03:35:22.468305 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.468202 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:35:22.477039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.477003 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.479005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.478976 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:35:22.479005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.478990 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.479005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.478990 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.490313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.490280 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj94l\" (UniqueName: \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\") pod \"maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.589006 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.588915 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:35:22.597005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.596972 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:22.679824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.679739 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:22.887698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:22.886722 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:35:22.890159 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:35:22.890128 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c6613_cca8_4eb8_95c5_1daebca4a49a.slice/crio-19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2 WatchSource:0}: Error finding container 19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2: Status 404 returned error can't find the container with id 19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2 Oct 31 03:35:23.253218 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.253179 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:23.767852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.767817 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" event={"ID":"3d4c6613-cca8-4eb8-95c5-1daebca4a49a","Type":"ContainerStarted","Data":"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4"} Oct 31 03:35:23.768276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.767875 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" event={"ID":"3d4c6613-cca8-4eb8-95c5-1daebca4a49a","Type":"ContainerStarted","Data":"19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2"} Oct 31 03:35:23.768276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.767905 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:23.799892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.799837 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" podStartSLOduration=1.320971377 podStartE2EDuration="1.799819665s" totalImagesPullingTime="478.848288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:23.798809635 +0000 UTC m=+2401.653826799" watchObservedRunningTime="2026-10-31 03:35:23.799819665 +0000 UTC m=+2401.654836884" Oct 31 03:35:23.893538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:23.893505 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:24.308583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:24.308538 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:35:24.763986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:24.763949 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.388389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:25.388346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.980449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:25.975990 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:26.652664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:26.652627 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:27.189111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:27.188999 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:27.874401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:27.874364 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:28.462752 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:28.462713 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:28.841711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:28.841627 4556 scope.go:122] "RemoveContainer" containerID="1492edfb786526110dd19720db7daa94a2e59829e3d316303d58ee149c17d0f4" Oct 31 03:35:28.850174 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:28.850151 4556 scope.go:122] "RemoveContainer" containerID="20c3399816a8239c40a49dcf3aaed6fedb661e3937ef2eca9f6d4774857ef433" Oct 31 03:35:28.861371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:28.861343 4556 scope.go:122] "RemoveContainer" containerID="84f798981149fc60b4e55d63573479bce5a5ddfb46255d1c2155f6d29299114d" Oct 31 03:35:28.895578 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:28.895553 4556 scope.go:122] "RemoveContainer" containerID="9a3caad7f166f2f6fb03a5736f7ef701b96d9d5d21dccc258b8ae403c3a8694d" Oct 31 03:35:29.156823 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:29.156779 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:29.693249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:29.693209 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:29.794580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:29.794545 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:35:30.390243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:30.390198 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:31.248801 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:31.248759 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:31.886360 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:31.886319 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:32.482846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:32.482807 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:33.166067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:33.166006 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:33.784840 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:33.784803 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:34.446846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.446792 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:34.747583 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.747547 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:34.747792 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.747616 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:34.844030 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.843989 4556 generic.go:356] "Generic (PLEG): container finished" podID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerID="288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87" exitCode=0 Oct 31 03:35:34.844230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.844064 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerDied","Data":"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87"} Oct 31 03:35:34.844497 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.844482 4556 scope.go:122] "RemoveContainer" containerID="288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87" Oct 31 03:35:34.845740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.845714 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:34.845809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.845783 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:34.845846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:34.845815 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:35:35.090556 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:35.090450 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.690437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:35.689533 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.851021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:35.850985 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerStarted","Data":"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99"} Oct 31 03:35:36.195456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:36.186760 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:36.460386 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:36.460302 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:36.460571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:36.460484 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:36.460571 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:36.460533 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:37.332312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:37.332263 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:37.460977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:37.460941 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:37.461172 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:37.460998 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:37.893050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:37.893012 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.022994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.022954 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:35:38.026699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.026675 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.033444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.033392 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-31e475\"/\"openshift-service-ca.crt\"" Oct 31 03:35:38.034160 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.033671 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-31e475\"/\"simulator-single-main-31e475-kserve-self-signed-certs\"" Oct 31 03:35:38.034257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.033807 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-31e475\"/\"kube-root-ca.crt\"" Oct 31 03:35:38.034257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.033989 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-31e475\"/\"default-dockercfg-rw5gl\"" Oct 31 03:35:38.038958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.038930 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:35:38.101602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.101555 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.139739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.139699 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.139910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.139752 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.139910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.139867 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.140009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.139987 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.140058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.140036 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.140093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.140059 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvth\" (UniqueName: \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240606 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240692 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240726 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240743 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvth\" (UniqueName: \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240779 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.240995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.240832 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.241151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.241124 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.241210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.241153 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.241311 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.241295 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.243117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.243093 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.243640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.243621 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.252280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.252222 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvth\" (UniqueName: \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\") pod \"simulator-single-main-31e475-kserve-795ff75b79-btzrc\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.339377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.339331 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:38.461135 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.461090 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:38.461320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.461157 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:38.635609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.635516 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:35:38.639555 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:35:38.639521 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb75b5f0_9bce_4f3a_8f18_60117a3c21a6.slice/crio-aad712c8437791936fae83a1d6c6da6c08f28ad1ab9b0e9a0bc7cbdcd3019a0a WatchSource:0}: Error finding container aad712c8437791936fae83a1d6c6da6c08f28ad1ab9b0e9a0bc7cbdcd3019a0a: Status 404 returned error can't find the container with id aad712c8437791936fae83a1d6c6da6c08f28ad1ab9b0e9a0bc7cbdcd3019a0a Oct 31 03:35:38.656354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.656310 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.867404 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.867361 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" event={"ID":"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6","Type":"ContainerStarted","Data":"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c"} Oct 31 03:35:38.867620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:38.867451 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" event={"ID":"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6","Type":"ContainerStarted","Data":"aad712c8437791936fae83a1d6c6da6c08f28ad1ab9b0e9a0bc7cbdcd3019a0a"} Oct 31 03:35:39.339786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.339752 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:39.461103 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.461057 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:39.461295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.461145 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:39.464535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.464499 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:39.763928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.763870 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" podStartSLOduration=2.7638497060000002 podStartE2EDuration="2.763849706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:38.898783567 +0000 UTC m=+2416.753800709" watchObservedRunningTime="2026-10-31 03:35:39.763849706 +0000 UTC m=+2417.618866849" Oct 31 03:35:39.765045 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:39.765015 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:35:40.408866 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:40.408831 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:40.460722 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:40.460678 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:35:40.460927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:40.460745 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:35:41.217267 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:41.217219 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.464987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:41.464953 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:41.465466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:41.465266 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:41.466147 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:41.466124 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:35:41.768759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:41.768669 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:42.477334 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:42.477300 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:43.213279 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:43.213241 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:43.763785 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:43.763723 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:44.422062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:44.422024 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.080268 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:45.080222 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.677872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:45.677838 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.936601 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:35:45.936515 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:35:46.546580 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:46.546545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:47.080980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:47.080943 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:47.780645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:47.780613 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:48.340286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:48.340247 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:48.353379 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:48.353347 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:48.446967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:48.446887 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:48.954043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:48.954007 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:35:49.071855 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:49.071822 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:49.789391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:49.789337 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:50.459969 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:50.459931 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:51.252669 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:51.252612 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:52.040095 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:52.040063 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:52.827204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:52.827167 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:53.463886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:53.463828 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:54.125687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:54.125636 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:54.736270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:54.736230 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.362628 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:55.362588 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.837821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:55.837784 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:35:55.996696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:55.996658 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:56.772974 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:56.772928 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:57.026997 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:57.026904 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6"] Oct 31 03:35:57.032152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:57.032120 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zzh6"] Oct 31 03:35:57.375646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:57.375529 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:57.971142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:57.971102 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:58.256215 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:58.256130 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b635038-1e88-4f65-9a72-4df732131665" path="/var/lib/kubelet/pods/8b635038-1e88-4f65-9a72-4df732131665/volumes" Oct 31 03:35:58.837684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:58.837645 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:59.750109 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:35:59.750036 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:00.359338 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:00.359298 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:00.975340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:00.975296 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:01.659666 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:01.659633 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:02.344618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:02.344575 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:02.474402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:02.474359 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:02.964742 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:02.964702 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:03.640888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:03.640851 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:04.268449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:04.268367 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:04.864280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:04.864247 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:05.467444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:05.467386 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:06.076209 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:06.076179 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:06.679493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:06.679445 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:07.367775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:07.367743 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:07.979125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:07.979079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:08.029086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:08.029038 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56"] Oct 31 03:36:08.032449 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:08.032405 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2pc56"] Oct 31 03:36:08.256296 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:08.256217 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e58b58a-3c6f-4499-a885-a8f8e953844b" path="/var/lib/kubelet/pods/3e58b58a-3c6f-4499-a885-a8f8e953844b/volumes" Oct 31 03:36:08.732591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:08.732560 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:09.473707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:09.473672 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:10.142429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:10.142372 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:10.308581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:10.308546 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:10.813105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:10.813069 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:10.968387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:10.968346 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:11.570313 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:11.570267 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:12.278700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:12.278662 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:12.891605 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:12.891571 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.476186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:13.476153 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.641735 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:13.641697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.779149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:13.779066 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:14.072291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:14.072201 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:14.667159 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:14.667120 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:15.307016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:15.306979 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:15.936552 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:15.936508 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:36:15.967250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:15.967094 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.590865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:16.590827 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:17.271186 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:17.271153 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:17.913573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:17.913534 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:18.552768 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:18.552708 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:19.166826 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:19.166756 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:19.385967 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:19.385929 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:19.624192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:19.624154 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:36:19.926668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:19.926631 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:20.657206 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:20.657169 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:22.272696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:22.272647 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:22.408048 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:22.408008 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:22.574246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:22.574162 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.171084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:23.171046 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.768766 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:23.768723 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.300124 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:24.300088 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:36:24.432028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:24.431988 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.578243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:24.578112 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:25.055994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:25.055934 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:25.661591 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:25.661557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:26.271609 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:26.271572 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:26.954721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:26.954679 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:27.559878 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:27.559817 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:28.178266 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:28.178225 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:28.772502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:28.772469 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:28.927587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:28.927552 4556 scope.go:122] "RemoveContainer" containerID="ab96cdfb69d0b6ccdffa062c2a95c27f5b4ef637612a7592cf6539d337f3724e" Oct 31 03:36:28.936653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:28.936630 4556 scope.go:122] "RemoveContainer" containerID="2de15c1372112ea50bd8b2123431801e8f056678d6e3ebf5043adc48aa3011a8" Oct 31 03:36:28.946020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:28.945998 4556 scope.go:122] "RemoveContainer" containerID="3b25864ac61b465700798d8e4b2cc17cdec135eda61e8c8518ba9f08996c47c3" Oct 31 03:36:29.497956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:29.497924 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.723216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:29.723182 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.856533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:29.856435 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.164760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.164717 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.590456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.590321 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:36:30.590655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.590591 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" podUID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" containerName="maas-api" containerID="cri-o://7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4" gracePeriod=30 Oct 31 03:36:30.733351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.733311 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.871632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.871599 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.873012 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.872994 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:36:30.909749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.909720 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\" (UniqueName: \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\") pod \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " Oct 31 03:36:30.909927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.909763 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\") pod \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " Oct 31 03:36:30.909927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.909863 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\") pod \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " Oct 31 03:36:30.910023 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.909954 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\") pod \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\" (UID: \"3d4c6613-cca8-4eb8-95c5-1daebca4a49a\") " Oct 31 03:36:30.910245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.910206 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca" pod "3d4c6613-cca8-4eb8-95c5-1daebca4a49a" (UID: "3d4c6613-cca8-4eb8-95c5-1daebca4a49a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:30.911979 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.911953 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls" pod "3d4c6613-cca8-4eb8-95c5-1daebca4a49a" (UID: "3d4c6613-cca8-4eb8-95c5-1daebca4a49a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:30.912084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.912010 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l" pod "3d4c6613-cca8-4eb8-95c5-1daebca4a49a" (UID: "3d4c6613-cca8-4eb8-95c5-1daebca4a49a"). InnerVolumeSpecName "kube-api-access-cj94l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:30.912084 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:30.912016 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs" pod "3d4c6613-cca8-4eb8-95c5-1daebca4a49a" (UID: "3d4c6613-cca8-4eb8-95c5-1daebca4a49a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:31.011337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.011292 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj94l\" (UniqueName: \"kubernetes.io/projected/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-kube-api-access-cj94l\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:31.011337 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.011332 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:31.011701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.011353 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:31.011701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.011367 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4c6613-cca8-4eb8-95c5-1daebca4a49a-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:31.013695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.013623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:31.174754 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.174720 4556 generic.go:356] "Generic (PLEG): container finished" podID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" containerID="7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4" exitCode=0 Oct 31 03:36:31.174958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.174769 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" event={"ID":"3d4c6613-cca8-4eb8-95c5-1daebca4a49a","Type":"ContainerDied","Data":"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4"} Oct 31 03:36:31.174958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.174797 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" Oct 31 03:36:31.174958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.174820 4556 scope.go:122] "RemoveContainer" containerID="7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4" Oct 31 03:36:31.174958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.174809 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7" event={"ID":"3d4c6613-cca8-4eb8-95c5-1daebca4a49a","Type":"ContainerDied","Data":"19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2"} Oct 31 03:36:31.186465 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.185497 4556 scope.go:122] "RemoveContainer" containerID="7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4" Oct 31 03:36:31.191853 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:31.191816 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4\": container with ID starting with 7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4 not found: ID does not exist" containerID="7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4" Oct 31 03:36:31.192021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.191868 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4"} err="failed to get container status \"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4\": rpc error: code = NotFound desc = could not find container \"7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4\": container with ID starting with 7e1954af12d3fb1abcdddd31c6c59b8b6475820ac18e93a8d4d76adf03858af4 not found: ID does not exist" Oct 31 03:36:31.205146 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:31.205103 4556 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c6613_cca8_4eb8_95c5_1daebca4a49a.slice/crio-19f067b0e6b279a26f142410e02dbc420e6ab7b3d3bde9b0a012c08f4baeb5b2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c6613_cca8_4eb8_95c5_1daebca4a49a.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:36:31.206597 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.206571 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:36:31.210258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.210229 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-31e475-77d4986dc9-cb6k7"] Oct 31 03:36:31.446656 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:31.446569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:32.255813 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:32.255779 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" path="/var/lib/kubelet/pods/3d4c6613-cca8-4eb8-95c5-1daebca4a49a/volumes" Oct 31 03:36:32.498213 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:32.498169 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:36:38.282991 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:38.282910 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:38.421621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:38.421583 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:38.628760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:38.628666 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:36:38.630595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:38.630526 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" containerID="cri-o://cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99" gracePeriod=30 Oct 31 03:36:38.709977 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:38.709938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:39.320852 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.320782 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" podUID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" containerName="main" containerID="cri-o://499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c" gracePeriod=45 Oct 31 03:36:39.574638 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.574565 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:36:39.698904 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.698866 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.698980 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699011 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699036 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699092 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699067 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\" (UniqueName: \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699131 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\") pod \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\" (UID: \"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6\") " Oct 31 03:36:39.699342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699309 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:39.699342 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699321 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:39.699534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699336 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:39.699594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699546 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:39.699594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699562 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:39.699594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.699577 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:39.701314 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.701285 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:39.701504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.701362 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "kube-api-access-2vvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:39.701504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.701427 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs" pod "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" (UID: "eb75b5f0-9bce-4f3a-8f18-60117a3c21a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:39.800789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.800742 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:39.800789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.800786 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvth\" (UniqueName: \"kubernetes.io/projected/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-kube-api-access-2vvth\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:39.800789 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:39.800797 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:40.220027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.219990 4556 generic.go:356] "Generic (PLEG): container finished" podID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" containerID="499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c" exitCode=0 Oct 31 03:36:40.220227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.220033 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" Oct 31 03:36:40.220227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.220079 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" event={"ID":"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6","Type":"ContainerDied","Data":"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c"} Oct 31 03:36:40.220227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.220120 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc" event={"ID":"eb75b5f0-9bce-4f3a-8f18-60117a3c21a6","Type":"ContainerDied","Data":"aad712c8437791936fae83a1d6c6da6c08f28ad1ab9b0e9a0bc7cbdcd3019a0a"} Oct 31 03:36:40.220227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.220138 4556 scope.go:122] "RemoveContainer" containerID="499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c" Oct 31 03:36:40.232105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.232083 4556 scope.go:122] "RemoveContainer" containerID="499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c" Oct 31 03:36:40.232480 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:40.232459 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c\": container with ID starting with 499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c not found: ID does not exist" containerID="499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c" Oct 31 03:36:40.232541 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.232492 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c"} err="failed to get container status \"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c\": rpc error: code = NotFound desc = could not find container \"499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c\": container with ID starting with 499fb7988935fecfd57f05fb237d474a13da33640ba23a70f17933d40c22344c not found: ID does not exist" Oct 31 03:36:40.243086 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.243055 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:36:40.247353 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.247326 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-31e475/simulator-single-main-31e475-kserve-795ff75b79-btzrc"] Oct 31 03:36:40.256820 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:40.256789 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" path="/var/lib/kubelet/pods/eb75b5f0-9bce-4f3a-8f18-60117a3c21a6/volumes" Oct 31 03:36:41.468273 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:41.468239 4556 patch_prober.go:29] interesting pod/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:36:41.468716 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:41.468294 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:36:43.236019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.235985 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:36:43.236475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236374 4556 cpu_manager.go:414] "containerMap: removing container" podUID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" containerName="maas-api" Oct 31 03:36:43.236475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236384 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" containerName="maas-api" Oct 31 03:36:43.236475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236403 4556 cpu_manager.go:414] "containerMap: removing container" podUID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" containerName="main" Oct 31 03:36:43.236475 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236409 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" containerName="main" Oct 31 03:36:43.236636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236479 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4c6613-cca8-4eb8-95c5-1daebca4a49a" containerName="maas-api" Oct 31 03:36:43.236636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.236490 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb75b5f0-9bce-4f3a-8f18-60117a3c21a6" containerName="main" Oct 31 03:36:43.240858 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.240831 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.251486 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.251458 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:36:43.282853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.282821 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-97623009-kserve-self-signed-certs\"" Oct 31 03:36:43.320208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.320174 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:43.334568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334534 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.334759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334686 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.334759 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334743 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.334842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334774 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.334842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334795 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.334929 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.334852 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8wg\" (UniqueName: \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435394 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435478 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435505 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435526 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435557 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8wg\" (UniqueName: \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.435646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.435604 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.436037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.436007 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.436037 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.436002 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.436227 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.436054 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.437980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.437956 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.438111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.438093 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.448049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.448015 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8wg\" (UniqueName: \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\") pod \"e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.556085 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.555990 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:43.844685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.844645 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:36:43.848285 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:36:43.848254 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb44ad1da_ba96_4282_b5ed_0220a4aef5fd.slice/crio-6ccc0ecd56daa2250ee114a4ce1e5f4d18f1ff6f5fc24dd97a0657045288c3bc WatchSource:0}: Error finding container 6ccc0ecd56daa2250ee114a4ce1e5f4d18f1ff6f5fc24dd97a0657045288c3bc: Status 404 returned error can't find the container with id 6ccc0ecd56daa2250ee114a4ce1e5f4d18f1ff6f5fc24dd97a0657045288c3bc Oct 31 03:36:43.891208 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.891175 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:36:43.942427 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942369 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942454 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942552 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942603 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942636 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942696 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942685 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942727 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942727 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:43.942992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942780 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.942992 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.942816 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\") pod \"54313f3f-f10e-4281-ae4b-c71f0112aec2\" (UID: \"54313f3f-f10e-4281-ae4b-c71f0112aec2\") " Oct 31 03:36:43.943269 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.943183 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:43.943587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.943517 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:43.943803 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.943739 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:43.944120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.944096 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:43.944312 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.944284 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:43.946259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.945292 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:36:43.946960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.946939 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "kube-api-access-48xxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:43.947223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.947205 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:43.947561 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:43.947544 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy" pod "54313f3f-f10e-4281-ae4b-c71f0112aec2" (UID: "54313f3f-f10e-4281-ae4b-c71f0112aec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:44.043800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043755 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.043800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043792 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.043800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043803 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.043800 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043811 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.044081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043820 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.044081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043828 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54313f3f-f10e-4281-ae4b-c71f0112aec2-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.044081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043837 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54313f3f-f10e-4281-ae4b-c71f0112aec2-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.044081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.043845 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48xxf\" (UniqueName: \"kubernetes.io/projected/54313f3f-f10e-4281-ae4b-c71f0112aec2-kube-api-access-48xxf\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.243026 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.242990 4556 generic.go:356] "Generic (PLEG): container finished" podID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerID="cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99" exitCode=0 Oct 31 03:36:44.243544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.243041 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" Oct 31 03:36:44.243544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.243075 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerDied","Data":"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99"} Oct 31 03:36:44.243544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.243167 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr" event={"ID":"54313f3f-f10e-4281-ae4b-c71f0112aec2","Type":"ContainerDied","Data":"e16b6ad0ced26865ff7ddd14ca5730777cefb3b1c963b494c5c77b344b3acc61"} Oct 31 03:36:44.243544 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.243200 4556 scope.go:122] "RemoveContainer" containerID="cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99" Oct 31 03:36:44.244910 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.244881 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" event={"ID":"b44ad1da-ba96-4282-b5ed-0220a4aef5fd","Type":"ContainerStarted","Data":"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556"} Oct 31 03:36:44.245056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.245038 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" event={"ID":"b44ad1da-ba96-4282-b5ed-0220a4aef5fd","Type":"ContainerStarted","Data":"6ccc0ecd56daa2250ee114a4ce1e5f4d18f1ff6f5fc24dd97a0657045288c3bc"} Oct 31 03:36:44.254516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.254491 4556 scope.go:122] "RemoveContainer" containerID="288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87" Oct 31 03:36:44.263961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.263937 4556 scope.go:122] "RemoveContainer" containerID="cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99" Oct 31 03:36:44.264292 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:44.264266 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99\": container with ID starting with cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99 not found: ID does not exist" containerID="cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99" Oct 31 03:36:44.264525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.264300 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99"} err="failed to get container status \"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99\": rpc error: code = NotFound desc = could not find container \"cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99\": container with ID starting with cac5c2c4f50d016794e7e35589ab75dc0fa21dc569f9473cfd4eeef2e1fd1a99 not found: ID does not exist" Oct 31 03:36:44.264525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.264324 4556 scope.go:122] "RemoveContainer" containerID="288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87" Oct 31 03:36:44.264677 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:44.264653 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87\": container with ID starting with 288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87 not found: ID does not exist" containerID="288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87" Oct 31 03:36:44.264744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.264723 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87"} err="failed to get container status \"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87\": rpc error: code = NotFound desc = could not find container \"288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87\": container with ID starting with 288c346ef9cc9ab17efa1fdeed7cea63356cc3feeedd84a4d1be807fef8bbc87 not found: ID does not exist" Oct 31 03:36:44.265242 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.265034 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" podStartSLOduration=1.2650187179999999 podStartE2EDuration="1.265018718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:44.264326627 +0000 UTC m=+2482.119343766" watchObservedRunningTime="2026-10-31 03:36:44.265018718 +0000 UTC m=+2482.120035858" Oct 31 03:36:44.278698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.278665 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:36:44.285398 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.285363 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-31e475-openshift-default-7f5dd97485tcsrr"] Oct 31 03:36:44.557094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.557006 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:44.813689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:44.813583 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:36:45.940646 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:36:45.940601 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:36:46.259098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:46.259010 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" path="/var/lib/kubelet/pods/54313f3f-f10e-4281-ae4b-c71f0112aec2/volumes" Oct 31 03:36:53.556681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:53.556640 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:53.569620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:53.569592 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:54.320492 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:54.320464 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:36:59.209645 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.209611 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:36:59.210058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.210007 4556 cpu_manager.go:414] "containerMap: removing container" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:36:59.210058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.210019 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:36:59.210152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.210098 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:36:59.213738 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.213713 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.223214 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.222068 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-97623009-kserve-self-signed-certs\"" Oct 31 03:36:59.225526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.225485 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:36:59.242479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242437 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.242672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242499 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.242672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242551 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.242672 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242630 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.242795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242745 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.242795 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.242786 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbs4\" (UniqueName: \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.317899 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.317862 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:59.343956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.343918 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.344234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344209 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.344372 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344354 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344603 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbs4\" (UniqueName: \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344740 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344774 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.344906 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.345209 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.345681 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.345639 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.350136 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.350072 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.352758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.352725 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.361191 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.361152 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbs4\" (UniqueName: \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\") pod \"e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.531548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.531443 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:36:59.810946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:36:59.810723 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:36:59.813844 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:36:59.813811 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c2e9b16_72ef_469b_8870_92509abb87c0.slice/crio-3cc0fd14d1dca0aa4e99171b75f91c59568df5011be5efc9e7280063cb50ad82 WatchSource:0}: Error finding container 3cc0fd14d1dca0aa4e99171b75f91c59568df5011be5efc9e7280063cb50ad82: Status 404 returned error can't find the container with id 3cc0fd14d1dca0aa4e99171b75f91c59568df5011be5efc9e7280063cb50ad82 Oct 31 03:37:00.321949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.321914 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:00.343005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.342969 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" event={"ID":"7c2e9b16-72ef-469b-8870-92509abb87c0","Type":"ContainerStarted","Data":"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb"} Oct 31 03:37:00.343005 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.343013 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" event={"ID":"7c2e9b16-72ef-469b-8870-92509abb87c0","Type":"ContainerStarted","Data":"3cc0fd14d1dca0aa4e99171b75f91c59568df5011be5efc9e7280063cb50ad82"} Oct 31 03:37:00.366655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.366591 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" podStartSLOduration=1.3665768759999999 podStartE2EDuration="1.366576876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:00.365524839 +0000 UTC m=+2498.220541981" watchObservedRunningTime="2026-10-31 03:37:00.366576876 +0000 UTC m=+2498.221594007" Oct 31 03:37:00.532548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.532501 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:00.844369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:00.844337 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:02.608089 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:02.608046 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:08.841200 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:08.841158 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:09.531846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:09.531806 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:09.544508 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:09.544473 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:10.106821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:10.106787 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:10.407216 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:10.407180 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:10.854567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:10.854484 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:15.935931 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:37:15.935890 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:37:16.801433 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.801377 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p"] Oct 31 03:37:16.801799 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.801786 4556 cpu_manager.go:414] "containerMap: removing container" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:37:16.801853 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.801800 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:37:16.801894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.801864 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="54313f3f-f10e-4281-ae4b-c71f0112aec2" containerName="istio-proxy" Oct 31 03:37:16.805326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.805306 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.808368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.808329 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:37:16.808880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.808860 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:37:16.808995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.808887 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:37:16.808995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.808890 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:37:16.814364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.814335 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p"] Oct 31 03:37:16.875982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.875942 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.876211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.876015 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkdg\" (UniqueName: \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.876211 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.876163 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.977746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.977702 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.978205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.977800 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkdg\" (UniqueName: \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.978205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.977872 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.978364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.978341 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.980256 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.980233 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:16.989693 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:16.989665 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkdg\" (UniqueName: \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\") pod \"maas-api-revoke-sub-452b6d7b-s8m5p\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:17.121001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:17.120910 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:17.398926 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:17.398897 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p"] Oct 31 03:37:17.402186 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:37:17.402155 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089382a9_1b28_454f_a2bf_e5f02138f0aa.slice/crio-f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229 WatchSource:0}: Error finding container f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229: Status 404 returned error can't find the container with id f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229 Oct 31 03:37:17.435657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:17.435622 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:17.437019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:17.436987 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" event={"ID":"089382a9-1b28-454f-a2bf-e5f02138f0aa","Type":"ContainerStarted","Data":"f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229"} Oct 31 03:37:18.445062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:18.444960 4556 generic.go:356] "Generic (PLEG): container finished" podID="089382a9-1b28-454f-a2bf-e5f02138f0aa" containerID="9d3ea914a2ad5b0d4127f70db34b5e7cd08a839e9d1217b25ec4f34fc70db7fc" exitCode=0 Oct 31 03:37:18.445062 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:18.445031 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" event={"ID":"089382a9-1b28-454f-a2bf-e5f02138f0aa","Type":"ContainerDied","Data":"9d3ea914a2ad5b0d4127f70db34b5e7cd08a839e9d1217b25ec4f34fc70db7fc"} Oct 31 03:37:19.859657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:19.859629 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:20.051980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.051879 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\" (UniqueName: \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\") pod \"089382a9-1b28-454f-a2bf-e5f02138f0aa\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " Oct 31 03:37:20.051980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.051966 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\") pod \"089382a9-1b28-454f-a2bf-e5f02138f0aa\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " Oct 31 03:37:20.052231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.052031 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\") pod \"089382a9-1b28-454f-a2bf-e5f02138f0aa\" (UID: \"089382a9-1b28-454f-a2bf-e5f02138f0aa\") " Oct 31 03:37:20.052324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.052288 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca" pod "089382a9-1b28-454f-a2bf-e5f02138f0aa" (UID: "089382a9-1b28-454f-a2bf-e5f02138f0aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:20.054151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.054123 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca" pod "089382a9-1b28-454f-a2bf-e5f02138f0aa" (UID: "089382a9-1b28-454f-a2bf-e5f02138f0aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:20.054270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.054169 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg" pod "089382a9-1b28-454f-a2bf-e5f02138f0aa" (UID: "089382a9-1b28-454f-a2bf-e5f02138f0aa"). InnerVolumeSpecName "kube-api-access-hvkdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:20.153177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.153131 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.153177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.153173 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089382a9-1b28-454f-a2bf-e5f02138f0aa-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.153177 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.153184 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkdg\" (UniqueName: \"kubernetes.io/projected/089382a9-1b28-454f-a2bf-e5f02138f0aa-kube-api-access-hvkdg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.460169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.460128 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" Oct 31 03:37:20.460387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.460125 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p" event={"ID":"089382a9-1b28-454f-a2bf-e5f02138f0aa","Type":"ContainerDied","Data":"f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229"} Oct 31 03:37:20.460387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:20.460256 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f469f90517dda332cd1aea07f975e000e6ec8db625b59cab5b4a2c12c3894229" Oct 31 03:37:27.718538 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:27.718500 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:37:27.801894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:27.801809 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:28.864941 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:28.864902 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:37:28.942369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:28.942336 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:28.979527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:28.979487 4556 scope.go:122] "RemoveContainer" containerID="f8638d5333224cd68bd1abe9393cefb1224114345100ff8995b6961ee983dc59" Oct 31 03:37:28.989525 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:28.989497 4556 scope.go:122] "RemoveContainer" containerID="be60a6672260ac97a514ebf83e13279be5797a67e2d3f2bb95f4f0082c01c3db" Oct 31 03:37:28.998913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:28.998887 4556 scope.go:122] "RemoveContainer" containerID="4911bfed025824a7d0e65558e600c95e1d5f44261e6956513ddbbe833145d709" Oct 31 03:37:29.307982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:29.307941 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:30.511758 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.511711 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp"] Oct 31 03:37:30.512291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.512120 4556 cpu_manager.go:414] "containerMap: removing container" podUID="089382a9-1b28-454f-a2bf-e5f02138f0aa" containerName="revoke-keys" Oct 31 03:37:30.512291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.512132 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089382a9-1b28-454f-a2bf-e5f02138f0aa" containerName="revoke-keys" Oct 31 03:37:30.512291 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.512227 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="089382a9-1b28-454f-a2bf-e5f02138f0aa" containerName="revoke-keys" Oct 31 03:37:30.517577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.517537 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.522804 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.522777 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:37:30.523035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.522794 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:37:30.523140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.522974 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:37:30.523201 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.523163 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:37:30.525965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.525929 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp"] Oct 31 03:37:30.544928 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.544881 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nm4x\" (UniqueName: \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.545132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.544988 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.545262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.545235 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.646362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.646299 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.646599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.646495 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.646599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.646550 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nm4x\" (UniqueName: \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.647396 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.647368 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.649102 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.649074 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.658749 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.658713 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nm4x\" (UniqueName: \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\") pod \"maas-api-revoke-sub-3e31a506-g68kp\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:30.844919 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:30.844806 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:31.249644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:31.249589 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp"] Oct 31 03:37:31.255866 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:37:31.255075 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afbc339_ced3_43ea_8d16_ef505603a2b9.slice/crio-94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38 WatchSource:0}: Error finding container 94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38: Status 404 returned error can't find the container with id 94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38 Oct 31 03:37:31.518139 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:31.517999 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" event={"ID":"4afbc339-ced3-43ea-8d16-ef505603a2b9","Type":"ContainerStarted","Data":"94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38"} Oct 31 03:37:32.545036 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:32.545002 4556 generic.go:356] "Generic (PLEG): container finished" podID="4afbc339-ced3-43ea-8d16-ef505603a2b9" containerID="161a283303cd9c5af58a2e1170870bb5761d02582af1368a154d0cea60fa103f" exitCode=0 Oct 31 03:37:32.545632 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:32.545484 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" event={"ID":"4afbc339-ced3-43ea-8d16-ef505603a2b9","Type":"ContainerDied","Data":"161a283303cd9c5af58a2e1170870bb5761d02582af1368a154d0cea60fa103f"} Oct 31 03:37:33.725994 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.725956 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:33.832627 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.832520 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\" (UniqueName: \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\") pod \"4afbc339-ced3-43ea-8d16-ef505603a2b9\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " Oct 31 03:37:33.832828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.832694 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\") pod \"4afbc339-ced3-43ea-8d16-ef505603a2b9\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " Oct 31 03:37:33.832828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.832768 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\") pod \"4afbc339-ced3-43ea-8d16-ef505603a2b9\" (UID: \"4afbc339-ced3-43ea-8d16-ef505603a2b9\") " Oct 31 03:37:33.834872 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.833216 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca" pod "4afbc339-ced3-43ea-8d16-ef505603a2b9" (UID: "4afbc339-ced3-43ea-8d16-ef505603a2b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:33.835487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.835456 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x" pod "4afbc339-ced3-43ea-8d16-ef505603a2b9" (UID: "4afbc339-ced3-43ea-8d16-ef505603a2b9"). InnerVolumeSpecName "kube-api-access-2nm4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:33.835595 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.835550 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca" pod "4afbc339-ced3-43ea-8d16-ef505603a2b9" (UID: "4afbc339-ced3-43ea-8d16-ef505603a2b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:33.933987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.933948 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nm4x\" (UniqueName: \"kubernetes.io/projected/4afbc339-ced3-43ea-8d16-ef505603a2b9-kube-api-access-2nm4x\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:33.933987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.933985 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:33.934204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:33.934001 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afbc339-ced3-43ea-8d16-ef505603a2b9-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:34.559140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:34.559054 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" Oct 31 03:37:34.559140 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:34.559087 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp" event={"ID":"4afbc339-ced3-43ea-8d16-ef505603a2b9","Type":"ContainerDied","Data":"94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38"} Oct 31 03:37:34.559344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:34.559163 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94766f1051cee039deffcf0807b1958569bd3c3e8cc822d4b933845e357b4e38" Oct 31 03:37:35.035440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:35.033866 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:41.409647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:41.409604 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:42.739377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:42.739310 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" podUID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" containerName="main" containerID="cri-o://bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556" gracePeriod=45 Oct 31 03:37:43.256074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.256041 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:37:43.354231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354123 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\" (UniqueName: \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.354231 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354187 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.354546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354299 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.354546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354330 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.354546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354355 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.354546 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.354378 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\") pod \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\" (UID: \"b44ad1da-ba96-4282-b5ed-0220a4aef5fd\") " Oct 31 03:37:43.355285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.355257 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:43.355436 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.355398 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:43.355523 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.355461 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:43.357310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.357285 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:43.357641 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.357622 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:43.358518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.358502 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg" pod "b44ad1da-ba96-4282-b5ed-0220a4aef5fd" (UID: "b44ad1da-ba96-4282-b5ed-0220a4aef5fd"). InnerVolumeSpecName "kube-api-access-fj8wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:43.455520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455458 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.455520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455492 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.455520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455505 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.455520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455530 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.455938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455544 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.455938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.455559 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj8wg\" (UniqueName: \"kubernetes.io/projected/b44ad1da-ba96-4282-b5ed-0220a4aef5fd-kube-api-access-fj8wg\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:43.626520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.618088 4556 generic.go:356] "Generic (PLEG): container finished" podID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" containerID="bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556" exitCode=0 Oct 31 03:37:43.626520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.618187 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" event={"ID":"b44ad1da-ba96-4282-b5ed-0220a4aef5fd","Type":"ContainerDied","Data":"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556"} Oct 31 03:37:43.626520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.618224 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" event={"ID":"b44ad1da-ba96-4282-b5ed-0220a4aef5fd","Type":"ContainerDied","Data":"6ccc0ecd56daa2250ee114a4ce1e5f4d18f1ff6f5fc24dd97a0657045288c3bc"} Oct 31 03:37:43.626520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.618249 4556 scope.go:122] "RemoveContainer" containerID="bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556" Oct 31 03:37:43.626520 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.618428 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2" Oct 31 03:37:43.643942 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.643883 4556 scope.go:122] "RemoveContainer" containerID="bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556" Oct 31 03:37:43.644317 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:37:43.644264 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556\": container with ID starting with bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556 not found: ID does not exist" containerID="bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556" Oct 31 03:37:43.644317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.644297 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556"} err="failed to get container status \"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556\": rpc error: code = NotFound desc = could not find container \"bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556\": container with ID starting with bb52455ec76f626a7a852f47752165d6280869ecfaec2935f98794cd12515556 not found: ID does not exist" Oct 31 03:37:43.652137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.652109 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:37:43.655844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.655816 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-97623009-kserve-5565ff4fd5-78jn2"] Oct 31 03:37:43.892284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:43.885515 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" podUID="7c2e9b16-72ef-469b-8870-92509abb87c0" containerName="main" containerID="cri-o://79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb" gracePeriod=45 Oct 31 03:37:44.205867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.205826 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:44.259667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.259627 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" path="/var/lib/kubelet/pods/b44ad1da-ba96-4282-b5ed-0220a4aef5fd/volumes" Oct 31 03:37:44.264671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264626 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.264864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264698 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.264864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264773 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.264864 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264834 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.265058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264864 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\" (UniqueName: \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.265058 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.264929 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\") pod \"7c2e9b16-72ef-469b-8870-92509abb87c0\" (UID: \"7c2e9b16-72ef-469b-8870-92509abb87c0\") " Oct 31 03:37:44.265184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.265112 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.265335 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.265314 4556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-tmp-dir\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.265503 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.265335 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.265585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.265513 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.267551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.267521 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.267551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.267524 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "kube-api-access-dvbs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:44.269179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.269150 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs" pod "7c2e9b16-72ef-469b-8870-92509abb87c0" (UID: "7c2e9b16-72ef-469b-8870-92509abb87c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:44.367032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.366962 4556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c2e9b16-72ef-469b-8870-92509abb87c0-tls-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.367032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.367032 4556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-dshm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.367032 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.367048 4556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-home\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.367397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.367064 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbs4\" (UniqueName: \"kubernetes.io/projected/7c2e9b16-72ef-469b-8870-92509abb87c0-kube-api-access-dvbs4\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.367397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.367082 4556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c2e9b16-72ef-469b-8870-92509abb87c0-model-cache\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.630286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.630186 4556 generic.go:356] "Generic (PLEG): container finished" podID="7c2e9b16-72ef-469b-8870-92509abb87c0" containerID="79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb" exitCode=0 Oct 31 03:37:44.635016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.630245 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" Oct 31 03:37:44.635016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.630252 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" event={"ID":"7c2e9b16-72ef-469b-8870-92509abb87c0","Type":"ContainerDied","Data":"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb"} Oct 31 03:37:44.635016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.630807 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7" event={"ID":"7c2e9b16-72ef-469b-8870-92509abb87c0","Type":"ContainerDied","Data":"3cc0fd14d1dca0aa4e99171b75f91c59568df5011be5efc9e7280063cb50ad82"} Oct 31 03:37:44.635016 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.630862 4556 scope.go:122] "RemoveContainer" containerID="79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb" Oct 31 03:37:44.653469 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.653444 4556 scope.go:122] "RemoveContainer" containerID="79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb" Oct 31 03:37:44.655021 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:37:44.654950 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb\": container with ID starting with 79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb not found: ID does not exist" containerID="79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb" Oct 31 03:37:44.655021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.654995 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb"} err="failed to get container status \"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb\": rpc error: code = NotFound desc = could not find container \"79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb\": container with ID starting with 79d968d8c77eec0ced62e18e0860cd2a504712fbbb037754e891eededd08aecb not found: ID does not exist" Oct 31 03:37:44.665444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.665382 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:37:44.693443 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:44.671503 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-97623009-kserve-5bc799d8f7-jqmz7"] Oct 31 03:37:45.290440 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:45.289585 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:37:45.971442 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:37:45.968610 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:37:46.256148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:46.256070 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2e9b16-72ef-469b-8870-92509abb87c0" path="/var/lib/kubelet/pods/7c2e9b16-72ef-469b-8870-92509abb87c0/volumes" Oct 31 03:37:48.827760 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.827725 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff"] Oct 31 03:37:48.828141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828090 4556 cpu_manager.go:414] "containerMap: removing container" podUID="7c2e9b16-72ef-469b-8870-92509abb87c0" containerName="main" Oct 31 03:37:48.828141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828102 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2e9b16-72ef-469b-8870-92509abb87c0" containerName="main" Oct 31 03:37:48.828141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828118 4556 cpu_manager.go:414] "containerMap: removing container" podUID="4afbc339-ced3-43ea-8d16-ef505603a2b9" containerName="revoke-keys" Oct 31 03:37:48.828141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828124 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afbc339-ced3-43ea-8d16-ef505603a2b9" containerName="revoke-keys" Oct 31 03:37:48.828281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828144 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" containerName="main" Oct 31 03:37:48.828281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828149 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" containerName="main" Oct 31 03:37:48.828281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828208 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2e9b16-72ef-469b-8870-92509abb87c0" containerName="main" Oct 31 03:37:48.828281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828218 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afbc339-ced3-43ea-8d16-ef505603a2b9" containerName="revoke-keys" Oct 31 03:37:48.828281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.828227 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44ad1da-ba96-4282-b5ed-0220a4aef5fd" containerName="main" Oct 31 03:37:48.832935 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.832906 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:48.837922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.837895 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:37:48.838132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.838114 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-54pxf\"" Oct 31 03:37:48.846811 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.843882 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff"] Oct 31 03:37:48.881527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.881487 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:37:48.881527 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.881508 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:37:48.956093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.956050 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:48.956319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.956128 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:48.956319 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:48.956204 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxw5h\" (UniqueName: \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.057070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.057031 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.057292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.057109 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.057292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.057142 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxw5h\" (UniqueName: \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.057927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.057900 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.059668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.059638 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.068918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.068895 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxw5h\" (UniqueName: \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\") pod \"maas-api-revoke-sub-058578c2-2w4ff\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.149112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.149033 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:49.424796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.424758 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff"] Oct 31 03:37:49.427810 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:37:49.427774 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4800db2_c612_46b0_9292_e12f5e87ccf0.slice/crio-b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e WatchSource:0}: Error finding container b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e: Status 404 returned error can't find the container with id b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e Oct 31 03:37:49.686243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.686211 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" event={"ID":"a4800db2-c612-46b0-9292-e12f5e87ccf0","Type":"ContainerStarted","Data":"8a3c582dcb4c3b2ab53cb1ca9fe45c686cea0064403a4fe2f59b3012abe130e2"} Oct 31 03:37:49.686407 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:49.686262 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" event={"ID":"a4800db2-c612-46b0-9292-e12f5e87ccf0","Type":"ContainerStarted","Data":"b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e"} Oct 31 03:37:50.693064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:50.693022 4556 generic.go:356] "Generic (PLEG): container finished" podID="a4800db2-c612-46b0-9292-e12f5e87ccf0" containerID="8a3c582dcb4c3b2ab53cb1ca9fe45c686cea0064403a4fe2f59b3012abe130e2" exitCode=0 Oct 31 03:37:50.693534 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:50.693108 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" event={"ID":"a4800db2-c612-46b0-9292-e12f5e87ccf0","Type":"ContainerDied","Data":"8a3c582dcb4c3b2ab53cb1ca9fe45c686cea0064403a4fe2f59b3012abe130e2"} Oct 31 03:37:51.840362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.840330 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:51.985142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.985042 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\" (UniqueName: \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\") pod \"a4800db2-c612-46b0-9292-e12f5e87ccf0\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " Oct 31 03:37:51.985142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.985095 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\") pod \"a4800db2-c612-46b0-9292-e12f5e87ccf0\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " Oct 31 03:37:51.985344 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.985203 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\") pod \"a4800db2-c612-46b0-9292-e12f5e87ccf0\" (UID: \"a4800db2-c612-46b0-9292-e12f5e87ccf0\") " Oct 31 03:37:51.985620 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.985590 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca" pod "a4800db2-c612-46b0-9292-e12f5e87ccf0" (UID: "a4800db2-c612-46b0-9292-e12f5e87ccf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:51.987317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.987289 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h" pod "a4800db2-c612-46b0-9292-e12f5e87ccf0" (UID: "a4800db2-c612-46b0-9292-e12f5e87ccf0"). InnerVolumeSpecName "kube-api-access-sxw5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:51.987408 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:51.987308 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca" pod "a4800db2-c612-46b0-9292-e12f5e87ccf0" (UID: "a4800db2-c612-46b0-9292-e12f5e87ccf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:52.086320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.086269 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxw5h\" (UniqueName: \"kubernetes.io/projected/a4800db2-c612-46b0-9292-e12f5e87ccf0-kube-api-access-sxw5h\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:52.086320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.086312 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-xks-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:52.086320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.086327 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4800db2-c612-46b0-9292-e12f5e87ccf0-maas-api-openshift-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:52.703480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.703440 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" Oct 31 03:37:52.703480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.703472 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff" event={"ID":"a4800db2-c612-46b0-9292-e12f5e87ccf0","Type":"ContainerDied","Data":"b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e"} Oct 31 03:37:52.703714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:52.703526 4556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98fabbc7118feaead9d8f4bf156d896f48fdb64e68876032e1d8a0a11d3864e" Oct 31 03:37:59.727856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:37:59.727816 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:38:01.049579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.049537 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:01.049993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.049960 4556 cpu_manager.go:414] "containerMap: removing container" podUID="a4800db2-c612-46b0-9292-e12f5e87ccf0" containerName="revoke-keys" Oct 31 03:38:01.049993 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.049976 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4800db2-c612-46b0-9292-e12f5e87ccf0" containerName="revoke-keys" Oct 31 03:38:01.050078 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.050049 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4800db2-c612-46b0-9292-e12f5e87ccf0" containerName="revoke-keys" Oct 31 03:38:01.053402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.053380 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.061001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.060788 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:38:01.061001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.060863 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 03:38:01.061001 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.060895 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vsq4\"" Oct 31 03:38:01.067019 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.066984 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:01.070181 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.070151 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8"] Oct 31 03:38:01.074105 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.074084 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.087381 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.087344 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8"] Oct 31 03:38:01.113685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.113652 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:38:01.113898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.113697 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:38:01.176009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.175972 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-metrics-certs\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.176009 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176009 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-maas-api-tls\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.176239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176130 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.176239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176193 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w758\" (UniqueName: \"kubernetes.io/projected/2c9225c1-75ee-4b76-9e11-06df9096f2fd-kube-api-access-2w758\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.176317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176242 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9w8\" (UniqueName: \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.176317 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176285 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.176401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176385 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9225c1-75ee-4b76-9e11-06df9096f2fd-openshift-service-ca\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.176484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.176408 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.277248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277203 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.277248 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277248 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w758\" (UniqueName: \"kubernetes.io/projected/2c9225c1-75ee-4b76-9e11-06df9096f2fd-kube-api-access-2w758\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277265 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9w8\" (UniqueName: \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277329 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277458 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9225c1-75ee-4b76-9e11-06df9096f2fd-openshift-service-ca\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277477 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277493 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-metrics-certs\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.277549 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.277510 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-maas-api-tls\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.278336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.278289 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9225c1-75ee-4b76-9e11-06df9096f2fd-openshift-service-ca\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.278336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.278323 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.280237 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.280213 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.280341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.280312 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-metrics-certs\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.280462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.280442 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.280567 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.280548 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9225c1-75ee-4b76-9e11-06df9096f2fd-maas-api-tls\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.288842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.288811 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9w8\" (UniqueName: \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\") pod \"maas-api-5ccbbc67c7-7qwdv\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.288978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.288846 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w758\" (UniqueName: \"kubernetes.io/projected/2c9225c1-75ee-4b76-9e11-06df9096f2fd-kube-api-access-2w758\") pod \"maas-api-57d9c7bd57-knch8\" (UID: \"2c9225c1-75ee-4b76-9e11-06df9096f2fd\") " pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.367434 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.367329 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:01.392859 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.392824 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:01.652376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.652348 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:01.655153 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:01.655123 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ea10c6_624b_4f2f_9f91_1756017090c8.slice/crio-dbce9a1b976d840fc966970c952b3c02d95e8011c100308acbf46463966aa918 WatchSource:0}: Error finding container dbce9a1b976d840fc966970c952b3c02d95e8011c100308acbf46463966aa918: Status 404 returned error can't find the container with id dbce9a1b976d840fc966970c952b3c02d95e8011c100308acbf46463966aa918 Oct 31 03:38:01.656487 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.656465 4556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:38:01.704281 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.704253 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8"] Oct 31 03:38:01.707025 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:01.706999 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9225c1_75ee_4b76_9e11_06df9096f2fd.slice/crio-ab112c885110ebe69bb6a514873cff3aa94acde995a0e2d58cf7556d1816fa9d WatchSource:0}: Error finding container ab112c885110ebe69bb6a514873cff3aa94acde995a0e2d58cf7556d1816fa9d: Status 404 returned error can't find the container with id ab112c885110ebe69bb6a514873cff3aa94acde995a0e2d58cf7556d1816fa9d Oct 31 03:38:01.747035 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.746997 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" event={"ID":"2c9225c1-75ee-4b76-9e11-06df9096f2fd","Type":"ContainerStarted","Data":"ab112c885110ebe69bb6a514873cff3aa94acde995a0e2d58cf7556d1816fa9d"} Oct 31 03:38:01.748286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:01.748263 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" event={"ID":"b1ea10c6-624b-4f2f-9f91-1756017090c8","Type":"ContainerStarted","Data":"dbce9a1b976d840fc966970c952b3c02d95e8011c100308acbf46463966aa918"} Oct 31 03:38:02.755274 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.755236 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" event={"ID":"b1ea10c6-624b-4f2f-9f91-1756017090c8","Type":"ContainerStarted","Data":"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81"} Oct 31 03:38:02.755822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.755316 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:02.757023 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.756990 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" event={"ID":"2c9225c1-75ee-4b76-9e11-06df9096f2fd","Type":"ContainerStarted","Data":"3adf6ecf560b016df4398e1acee4352555876319c9c3985fbee972ceba7084bf"} Oct 31 03:38:02.757165 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.757047 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:02.774246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.774191 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" podStartSLOduration=1.298700239 podStartE2EDuration="1.774176078s" totalImagesPullingTime="475.475839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:02.773761473 +0000 UTC m=+2560.628778615" watchObservedRunningTime="2026-10-31 03:38:02.774176078 +0000 UTC m=+2560.629193218" Oct 31 03:38:02.793456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:02.793365 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" podStartSLOduration=1.3330871819999999 podStartE2EDuration="1.793346222s" totalImagesPullingTime="460.25904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:02.79223342 +0000 UTC m=+2560.647250561" watchObservedRunningTime="2026-10-31 03:38:02.793346222 +0000 UTC m=+2560.648363363" Oct 31 03:38:06.044240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:06.044150 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn"] Oct 31 03:38:06.047286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:06.047254 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5w5qn"] Oct 31 03:38:06.255547 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:06.255505 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3" path="/var/lib/kubelet/pods/ba9e6a66-c6d7-4889-87c2-8b95d6dfb1f3/volumes" Oct 31 03:38:08.226765 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.226731 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:08.667613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.667577 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:08.774033 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.773986 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:08.774400 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.774371 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57d9c7bd57-knch8" Oct 31 03:38:08.822156 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.822112 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:08.850793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.850757 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:08.851014 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:08.850927 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" podUID="b1ea10c6-624b-4f2f-9f91-1756017090c8" containerName="maas-api" containerID="cri-o://c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81" gracePeriod=30 Oct 31 03:38:09.110539 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.110511 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:09.154602 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.154557 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\" (UniqueName: \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\") pod \"b1ea10c6-624b-4f2f-9f91-1756017090c8\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " Oct 31 03:38:09.154841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.154640 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\") pod \"b1ea10c6-624b-4f2f-9f91-1756017090c8\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " Oct 31 03:38:09.154841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.154714 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\") pod \"b1ea10c6-624b-4f2f-9f91-1756017090c8\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " Oct 31 03:38:09.154841 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.154800 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\") pod \"b1ea10c6-624b-4f2f-9f91-1756017090c8\" (UID: \"b1ea10c6-624b-4f2f-9f91-1756017090c8\") " Oct 31 03:38:09.155363 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.155321 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca" pod "b1ea10c6-624b-4f2f-9f91-1756017090c8" (UID: "b1ea10c6-624b-4f2f-9f91-1756017090c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:09.156970 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.156941 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs" pod "b1ea10c6-624b-4f2f-9f91-1756017090c8" (UID: "b1ea10c6-624b-4f2f-9f91-1756017090c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:09.157128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.157036 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8" pod "b1ea10c6-624b-4f2f-9f91-1756017090c8" (UID: "b1ea10c6-624b-4f2f-9f91-1756017090c8"). InnerVolumeSpecName "kube-api-access-nm9w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:09.157128 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.157058 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls" pod "b1ea10c6-624b-4f2f-9f91-1756017090c8" (UID: "b1ea10c6-624b-4f2f-9f91-1756017090c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:09.256117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.256032 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9w8\" (UniqueName: \"kubernetes.io/projected/b1ea10c6-624b-4f2f-9f91-1756017090c8-kube-api-access-nm9w8\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:09.256117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.256066 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:09.256117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.256077 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ea10c6-624b-4f2f-9f91-1756017090c8-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:09.256117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.256086 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ea10c6-624b-4f2f-9f91-1756017090c8-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:09.793153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.793117 4556 generic.go:356] "Generic (PLEG): container finished" podID="b1ea10c6-624b-4f2f-9f91-1756017090c8" containerID="c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81" exitCode=0 Oct 31 03:38:09.793153 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.793157 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" event={"ID":"b1ea10c6-624b-4f2f-9f91-1756017090c8","Type":"ContainerDied","Data":"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81"} Oct 31 03:38:09.793393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.793160 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" Oct 31 03:38:09.793393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.793201 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv" event={"ID":"b1ea10c6-624b-4f2f-9f91-1756017090c8","Type":"ContainerDied","Data":"dbce9a1b976d840fc966970c952b3c02d95e8011c100308acbf46463966aa918"} Oct 31 03:38:09.793393 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.793223 4556 scope.go:122] "RemoveContainer" containerID="c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81" Oct 31 03:38:09.802594 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.802572 4556 scope.go:122] "RemoveContainer" containerID="c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81" Oct 31 03:38:09.802909 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:09.802887 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81\": container with ID starting with c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81 not found: ID does not exist" containerID="c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81" Oct 31 03:38:09.802959 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.802920 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81"} err="failed to get container status \"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81\": rpc error: code = NotFound desc = could not find container \"c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81\": container with ID starting with c1da20a0eb51abc4b68f8a7315c519f8c78b54c9a7eb0afad07c897ed7cd9b81 not found: ID does not exist" Oct 31 03:38:09.816973 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.816935 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:09.830599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:09.823827 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccbbc67c7-7qwdv"] Oct 31 03:38:10.174263 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:10.174228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:10.256230 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:10.256197 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ea10c6-624b-4f2f-9f91-1756017090c8" path="/var/lib/kubelet/pods/b1ea10c6-624b-4f2f-9f91-1756017090c8/volumes" Oct 31 03:38:11.803046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:11.803009 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:11.938971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:11.938939 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:12.960725 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.960688 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:12.961297 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.961275 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b1ea10c6-624b-4f2f-9f91-1756017090c8" containerName="maas-api" Oct 31 03:38:12.961385 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.961300 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ea10c6-624b-4f2f-9f91-1756017090c8" containerName="maas-api" Oct 31 03:38:12.961480 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.961402 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ea10c6-624b-4f2f-9f91-1756017090c8" containerName="maas-api" Oct 31 03:38:12.965805 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.965778 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:12.965981 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.965924 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:12.969618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.969593 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:12.972697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.972672 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:12.973087 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.973064 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-17e11a7f-serving-cert\"" Oct 31 03:38:12.990718 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.990633 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-17e11a7f\"" Oct 31 03:38:12.993562 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.993529 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:12.996966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.996938 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:12.997357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.997333 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzd99\" (UniqueName: \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:12.997577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:12.997560 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:13.001678 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.001655 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.030151 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.030115 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:13.036690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.036635 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.073387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.073356 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-17e11a7f-dockercfg-js79j\"" Oct 31 03:38:13.098093 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098060 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbs5\" (UniqueName: \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.098289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098099 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.098289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098126 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.098289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098269 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.098457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098303 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.098457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098336 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzkfp\" (UniqueName: \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.098537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098495 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzd99\" (UniqueName: \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.098537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098519 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.098921 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.098899 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.109906 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.109868 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzd99\" (UniqueName: \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\") pod \"payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.184540 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.184506 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.200090 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200032 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.200308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200106 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbs5\" (UniqueName: \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.200308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200137 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.200308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200232 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.200308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200270 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.200587 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.200315 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzkfp\" (UniqueName: \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.202535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.202220 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.202535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.202339 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.204802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.204777 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.220769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.207982 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.220769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.213241 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbs5\" (UniqueName: \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\") pod \"payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.220769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.215274 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzkfp\" (UniqueName: \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\") pod \"maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.300806 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.300770 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.313245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.313219 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:13.318143 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.318076 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.320184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.320156 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.604825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.604786 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:13.608179 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:13.608136 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890fbaab_330e_4707_96d7_28f6cfdc989d.slice/crio-6768d17c638813fe6e8c8c760d6d4a8f70d7ef74d42c1b52d1baf2a63e8e302c WatchSource:0}: Error finding container 6768d17c638813fe6e8c8c760d6d4a8f70d7ef74d42c1b52d1baf2a63e8e302c: Status 404 returned error can't find the container with id 6768d17c638813fe6e8c8c760d6d4a8f70d7ef74d42c1b52d1baf2a63e8e302c Oct 31 03:38:13.651097 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.651074 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:13.653548 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.653519 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:13.654013 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:13.653984 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01f1bf2_e7a8_42f7_93e7_7bfeec237c42.slice/crio-8e010509c695a6dd6aef0f703b2f2c23d86629c473ff23f64015d5581e77b579 WatchSource:0}: Error finding container 8e010509c695a6dd6aef0f703b2f2c23d86629c473ff23f64015d5581e77b579: Status 404 returned error can't find the container with id 8e010509c695a6dd6aef0f703b2f2c23d86629c473ff23f64015d5581e77b579 Oct 31 03:38:13.654878 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:13.654854 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10471b75_3df4_4ca7_8e07_ca9a2086b7ba.slice/crio-2124ce9880b95ca29bbbeefeae90313a3e515addbb8e8e415aafb47d3ee94dad WatchSource:0}: Error finding container 2124ce9880b95ca29bbbeefeae90313a3e515addbb8e8e415aafb47d3ee94dad: Status 404 returned error can't find the container with id 2124ce9880b95ca29bbbeefeae90313a3e515addbb8e8e415aafb47d3ee94dad Oct 31 03:38:13.814504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.814462 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" event={"ID":"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42","Type":"ContainerStarted","Data":"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d"} Oct 31 03:38:13.814689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.814539 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:13.814689 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.814560 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" event={"ID":"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42","Type":"ContainerStarted","Data":"8e010509c695a6dd6aef0f703b2f2c23d86629c473ff23f64015d5581e77b579"} Oct 31 03:38:13.816489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.816450 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" event={"ID":"890fbaab-330e-4707-96d7-28f6cfdc989d","Type":"ContainerStarted","Data":"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5"} Oct 31 03:38:13.816646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.816511 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" event={"ID":"890fbaab-330e-4707-96d7-28f6cfdc989d","Type":"ContainerStarted","Data":"6768d17c638813fe6e8c8c760d6d4a8f70d7ef74d42c1b52d1baf2a63e8e302c"} Oct 31 03:38:13.816646 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.816553 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:13.817815 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.817794 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" event={"ID":"10471b75-3df4-4ca7-8e07-ca9a2086b7ba","Type":"ContainerStarted","Data":"2124ce9880b95ca29bbbeefeae90313a3e515addbb8e8e415aafb47d3ee94dad"} Oct 31 03:38:13.834565 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.834506 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" podStartSLOduration=1.834487821 podStartE2EDuration="1.834487821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:13.831592376 +0000 UTC m=+2571.686609517" watchObservedRunningTime="2026-10-31 03:38:13.834487821 +0000 UTC m=+2571.689504965" Oct 31 03:38:13.849927 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.849856 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" podStartSLOduration=1.849835255 podStartE2EDuration="1.849835255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:13.847897425 +0000 UTC m=+2571.702914567" watchObservedRunningTime="2026-10-31 03:38:13.849835255 +0000 UTC m=+2571.704852396" Oct 31 03:38:13.868351 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:13.868273 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:14.661489 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:14.661439 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:14.823881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:14.823841 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" event={"ID":"10471b75-3df4-4ca7-8e07-ca9a2086b7ba","Type":"ContainerStarted","Data":"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b"} Oct 31 03:38:14.824137 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:14.824116 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:14.843094 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:14.843046 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" podStartSLOduration=2.291962771 podStartE2EDuration="2.843034364s" totalImagesPullingTime="551.071593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:14.84258685 +0000 UTC m=+2572.697603992" watchObservedRunningTime="2026-10-31 03:38:14.843034364 +0000 UTC m=+2572.698051515" Oct 31 03:38:15.265115 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:15.265079 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:15.651662 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:15.651625 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:38:15.937509 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:15.937428 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:38:15.965699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:15.965663 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:16.693733 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:16.693697 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:17.396990 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:17.396952 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:18.086020 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:18.085982 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:18.882596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:18.882523 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.576671 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:19.576628 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:20.272695 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:20.272643 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:20.838462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:20.838406 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:20.985257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:20.981828 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:21.658600 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:21.658569 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:22.606002 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:22.605968 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:23.309336 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:23.309301 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:23.969049 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:23.969006 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:24.744262 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:24.744228 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:24.827999 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:24.827967 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:24.828199 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:24.828033 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:25.372604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:25.372572 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.080741 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:26.078962 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.886913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:26.886870 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:27.663664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:27.663623 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:28.674966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:28.674928 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:29.382171 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:29.382058 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:30.089368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:30.089334 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:30.882946 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:30.882916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:31.652450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:31.652379 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:32.269647 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:32.269607 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:32.858644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:32.858582 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:33.522258 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:33.522222 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:34.064771 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:34.064729 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:34.668581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:34.668538 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:35.275310 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:35.275270 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:35.976255 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:35.976212 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:36.573690 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:36.573653 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:37.159502 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:37.159462 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:37.805340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:37.805309 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:37.892596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:37.892555 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:37.903253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:37.903217 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:37.995510 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:37.995479 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.133633 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.133551 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.373264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.373226 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:38.373570 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.373517 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" podUID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" containerName="maas-api" containerID="cri-o://f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b" gracePeriod=30 Oct 31 03:38:38.457961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.457928 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.594786 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.594747 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.644027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.643639 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:38.660920 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.660885 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\" (UniqueName: \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\") pod \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " Oct 31 03:38:38.661134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.660952 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\") pod \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " Oct 31 03:38:38.661134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.660974 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\") pod \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " Oct 31 03:38:38.661134 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.661019 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\") pod \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\" (UID: \"10471b75-3df4-4ca7-8e07-ca9a2086b7ba\") " Oct 31 03:38:38.661868 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.661373 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca" pod "10471b75-3df4-4ca7-8e07-ca9a2086b7ba" (UID: "10471b75-3df4-4ca7-8e07-ca9a2086b7ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:38.666938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.663479 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs" pod "10471b75-3df4-4ca7-8e07-ca9a2086b7ba" (UID: "10471b75-3df4-4ca7-8e07-ca9a2086b7ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:38.666938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.663766 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls" pod "10471b75-3df4-4ca7-8e07-ca9a2086b7ba" (UID: "10471b75-3df4-4ca7-8e07-ca9a2086b7ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:38.666938 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.664898 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp" pod "10471b75-3df4-4ca7-8e07-ca9a2086b7ba" (UID: "10471b75-3df4-4ca7-8e07-ca9a2086b7ba"). InnerVolumeSpecName "kube-api-access-fzkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:38.762260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.762208 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzkfp\" (UniqueName: \"kubernetes.io/projected/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-kube-api-access-fzkfp\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:38.762260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.762251 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:38.762260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.762268 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:38.762673 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.762280 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10471b75-3df4-4ca7-8e07-ca9a2086b7ba-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:38.767493 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.767457 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.950292 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.950257 4556 generic.go:356] "Generic (PLEG): container finished" podID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" containerID="f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b" exitCode=0 Oct 31 03:38:38.950832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.950303 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" event={"ID":"10471b75-3df4-4ca7-8e07-ca9a2086b7ba","Type":"ContainerDied","Data":"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b"} Oct 31 03:38:38.950832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.950311 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" Oct 31 03:38:38.950832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.950343 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr" event={"ID":"10471b75-3df4-4ca7-8e07-ca9a2086b7ba","Type":"ContainerDied","Data":"2124ce9880b95ca29bbbeefeae90313a3e515addbb8e8e415aafb47d3ee94dad"} Oct 31 03:38:38.950832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.950364 4556 scope.go:122] "RemoveContainer" containerID="f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b" Oct 31 03:38:38.960238 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.960216 4556 scope.go:122] "RemoveContainer" containerID="f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b" Oct 31 03:38:38.960556 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:38.960535 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b\": container with ID starting with f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b not found: ID does not exist" containerID="f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b" Oct 31 03:38:38.960621 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.960566 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b"} err="failed to get container status \"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b\": rpc error: code = NotFound desc = could not find container \"f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b\": container with ID starting with f44e75ee558a8f68dbc7189184cd068b17cd1e7bf6dba549cb7732203573af9b not found: ID does not exist" Oct 31 03:38:38.971653 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.971620 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:38.974896 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:38.974868 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17e11a7f-7554d78c6d-jh7fr"] Oct 31 03:38:39.837253 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:39.837212 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:38:40.255721 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:40.255680 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" path="/var/lib/kubelet/pods/10471b75-3df4-4ca7-8e07-ca9a2086b7ba/volumes" Oct 31 03:38:42.893345 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:42.893300 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" podUID="890fbaab-330e-4707-96d7-28f6cfdc989d" containerName="payload-processing" containerID="cri-o://ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5" gracePeriod=25 Oct 31 03:38:42.903995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:42.903948 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" podUID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" containerName="payload-pre-processing" containerID="cri-o://70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d" gracePeriod=25 Oct 31 03:38:43.161846 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.161814 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:43.196797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.196761 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:43.206694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.206659 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\") pod \"890fbaab-330e-4707-96d7-28f6cfdc989d\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " Oct 31 03:38:43.206880 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.206784 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\" (UniqueName: \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\") pod \"890fbaab-330e-4707-96d7-28f6cfdc989d\" (UID: \"890fbaab-330e-4707-96d7-28f6cfdc989d\") " Oct 31 03:38:43.207116 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.207092 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume" pod "890fbaab-330e-4707-96d7-28f6cfdc989d" (UID: "890fbaab-330e-4707-96d7-28f6cfdc989d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:43.209046 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.209019 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99" pod "890fbaab-330e-4707-96d7-28f6cfdc989d" (UID: "890fbaab-330e-4707-96d7-28f6cfdc989d"). InnerVolumeSpecName "kube-api-access-qzd99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:43.308197 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.308160 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\" (UniqueName: \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\") pod \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " Oct 31 03:38:43.308444 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.308263 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\") pod \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\" (UID: \"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42\") " Oct 31 03:38:43.308535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.308465 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzd99\" (UniqueName: \"kubernetes.io/projected/890fbaab-330e-4707-96d7-28f6cfdc989d-kube-api-access-qzd99\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:43.308535 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.308479 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890fbaab-330e-4707-96d7-28f6cfdc989d-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:43.308744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.308718 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume" pod "b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" (UID: "b01f1bf2-e7a8-42f7-93e7-7bfeec237c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:43.310285 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.310263 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5" pod "b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" (UID: "b01f1bf2-e7a8-42f7-93e7-7bfeec237c42"). InnerVolumeSpecName "kube-api-access-6cbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:43.409187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.409143 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbs5\" (UniqueName: \"kubernetes.io/projected/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-kube-api-access-6cbs5\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:43.409187 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.409186 4556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42-plugins-config-volume\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:43.976668 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.976626 4556 generic.go:356] "Generic (PLEG): container finished" podID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" containerID="70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d" exitCode=0 Oct 31 03:38:43.977189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.976694 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" Oct 31 03:38:43.977189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.976705 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" event={"ID":"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42","Type":"ContainerDied","Data":"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d"} Oct 31 03:38:43.977189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.976802 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5" event={"ID":"b01f1bf2-e7a8-42f7-93e7-7bfeec237c42","Type":"ContainerDied","Data":"8e010509c695a6dd6aef0f703b2f2c23d86629c473ff23f64015d5581e77b579"} Oct 31 03:38:43.977189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.976838 4556 scope.go:122] "RemoveContainer" containerID="70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d" Oct 31 03:38:43.978357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.978180 4556 generic.go:356] "Generic (PLEG): container finished" podID="890fbaab-330e-4707-96d7-28f6cfdc989d" containerID="ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5" exitCode=0 Oct 31 03:38:43.978357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.978252 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" Oct 31 03:38:43.978357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.978269 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" event={"ID":"890fbaab-330e-4707-96d7-28f6cfdc989d","Type":"ContainerDied","Data":"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5"} Oct 31 03:38:43.978357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.978317 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp" event={"ID":"890fbaab-330e-4707-96d7-28f6cfdc989d","Type":"ContainerDied","Data":"6768d17c638813fe6e8c8c760d6d4a8f70d7ef74d42c1b52d1baf2a63e8e302c"} Oct 31 03:38:43.987714 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.987682 4556 scope.go:122] "RemoveContainer" containerID="70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d" Oct 31 03:38:43.988051 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:43.988023 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d\": container with ID starting with 70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d not found: ID does not exist" containerID="70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d" Oct 31 03:38:43.988122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.988066 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d"} err="failed to get container status \"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d\": rpc error: code = NotFound desc = could not find container \"70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d\": container with ID starting with 70098c8b58a4e745d341647047b1ba7178f67e43af8bd6cf1bae91764871619d not found: ID does not exist" Oct 31 03:38:43.988122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.988091 4556 scope.go:122] "RemoveContainer" containerID="ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5" Oct 31 03:38:43.997886 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.997859 4556 scope.go:122] "RemoveContainer" containerID="ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5" Oct 31 03:38:43.998194 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:43.998174 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5\": container with ID starting with ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5 not found: ID does not exist" containerID="ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5" Oct 31 03:38:43.998271 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:43.998203 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5"} err="failed to get container status \"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5\": rpc error: code = NotFound desc = could not find container \"ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5\": container with ID starting with ee1aeb584a67a25c264815af6c88cdd1536ee28e3689b4bbc92e34a8fac493e5 not found: ID does not exist" Oct 31 03:38:44.001816 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.001792 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:44.006857 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.006833 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17e11a7f-745659cbd5-kgqmp"] Oct 31 03:38:44.015965 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.015934 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:44.020438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.020396 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17e11a7f-55497c945f-vsgn5"] Oct 31 03:38:44.240975 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.240885 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:44.260308 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.260262 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890fbaab-330e-4707-96d7-28f6cfdc989d" path="/var/lib/kubelet/pods/890fbaab-330e-4707-96d7-28f6cfdc989d/volumes" Oct 31 03:38:44.260945 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:44.260918 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" path="/var/lib/kubelet/pods/b01f1bf2-e7a8-42f7-93e7-7bfeec237c42/volumes" Oct 31 03:38:45.589246 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.589210 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:45.943645 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:38:45.943597 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:38:45.955655 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.955611 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:38:45.956293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956269 4556 cpu_manager.go:414] "containerMap: removing container" podUID="890fbaab-330e-4707-96d7-28f6cfdc989d" containerName="payload-processing" Oct 31 03:38:45.956376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956298 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890fbaab-330e-4707-96d7-28f6cfdc989d" containerName="payload-processing" Oct 31 03:38:45.956376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956326 4556 cpu_manager.go:414] "containerMap: removing container" podUID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" containerName="maas-api" Oct 31 03:38:45.956376 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956335 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" containerName="maas-api" Oct 31 03:38:45.956554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956380 4556 cpu_manager.go:414] "containerMap: removing container" podUID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" containerName="payload-pre-processing" Oct 31 03:38:45.956554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956388 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" containerName="payload-pre-processing" Oct 31 03:38:45.956554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956548 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="890fbaab-330e-4707-96d7-28f6cfdc989d" containerName="payload-processing" Oct 31 03:38:45.956684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956567 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="10471b75-3df4-4ca7-8e07-ca9a2086b7ba" containerName="maas-api" Oct 31 03:38:45.956684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.956583 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01f1bf2-e7a8-42f7-93e7-7bfeec237c42" containerName="payload-pre-processing" Oct 31 03:38:45.961847 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.961820 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:45.972663 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.972638 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-8d20d2-openshift-default-dockercfg-pxx2c\"" Oct 31 03:38:45.983222 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:45.983195 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:38:46.035047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035006 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035047 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035144 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035208 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035276 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035240 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035304 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035373 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035400 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.035488 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.035451 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmlm\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.109529 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.109490 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:46.136978 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.136929 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137008 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmlm\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137064 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137095 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137189 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137134 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137233 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137273 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137455 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137300 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137640 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137457 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137703 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137684 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137705 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137918 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137893 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.137995 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.137939 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.138074 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.138046 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.139707 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.139672 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.139958 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.139938 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.148512 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.148482 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.148717 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.148693 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmlm\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\") pod \"e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.278777 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.278679 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:46.487125 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.487058 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:46.560288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.560264 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:38:46.563028 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:46.562994 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc2d605_d2f7_48ed_bbba_99620489dd31.slice/crio-634a16125659b95211938824ae572a7769cf6f306e7197e1b27ba429bdf86fef WatchSource:0}: Error finding container 634a16125659b95211938824ae572a7769cf6f306e7197e1b27ba429bdf86fef: Status 404 returned error can't find the container with id 634a16125659b95211938824ae572a7769cf6f306e7197e1b27ba429bdf86fef Oct 31 03:38:46.565430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.565382 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:38:46.565526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.565461 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:38:46.565526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:46.565491 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:38:47.000067 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:47.000024 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerStarted","Data":"7478856b4df5bc1c80b867db548f246555ef18b67daa71279cd0547b1579c537"} Oct 31 03:38:47.000495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:47.000095 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerStarted","Data":"634a16125659b95211938824ae572a7769cf6f306e7197e1b27ba429bdf86fef"} Oct 31 03:38:47.021659 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:47.021597 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podStartSLOduration=2.021579321 podStartE2EDuration="2.021579321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:47.020079143 +0000 UTC m=+2604.875096297" watchObservedRunningTime="2026-10-31 03:38:47.021579321 +0000 UTC m=+2604.876596464" Oct 31 03:38:47.279860 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:47.279772 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:47.285340 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:47.285310 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:48.005157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:48.005126 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:48.006341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:48.006322 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:38:49.329029 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:49.328997 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:49.462962 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:49.462923 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.473112 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.473072 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:38:50.477397 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.477366 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.481664 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.481633 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-8d20d2-serving-cert\"" Oct 31 03:38:50.492387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.492347 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:38:50.515448 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.514868 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.593126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.593092 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.593324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.593142 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.593324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.593298 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.593467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.593356 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdt7\" (UniqueName: \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.663925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.663874 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.694778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.694732 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.695007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.694804 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdt7\" (UniqueName: \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.695007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.694867 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.695007 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.694937 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.695832 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.695780 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.697719 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.697699 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.698192 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.698168 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.705914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.705883 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdt7\" (UniqueName: \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\") pod \"maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:50.795322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.795242 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.798286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:50.798270 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:51.076383 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:51.076302 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:38:51.079877 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:38:51.079848 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93595565_ed34_4e71_ba21_1a189f2901ef.slice/crio-d7dd74b777f81e7f756419297c62ec6baf390285a718c5441d2ea2fc70bdc2bd WatchSource:0}: Error finding container d7dd74b777f81e7f756419297c62ec6baf390285a718c5441d2ea2fc70bdc2bd: Status 404 returned error can't find the container with id d7dd74b777f81e7f756419297c62ec6baf390285a718c5441d2ea2fc70bdc2bd Oct 31 03:38:51.477129 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:51.477091 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:52.025307 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.025269 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" event={"ID":"93595565-ed34-4e71-ba21-1a189f2901ef","Type":"ContainerStarted","Data":"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619"} Oct 31 03:38:52.025581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.025329 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" event={"ID":"93595565-ed34-4e71-ba21-1a189f2901ef","Type":"ContainerStarted","Data":"d7dd74b777f81e7f756419297c62ec6baf390285a718c5441d2ea2fc70bdc2bd"} Oct 31 03:38:52.025581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.025407 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:52.046357 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.046299 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" podStartSLOduration=1.625498791 podStartE2EDuration="2.046279651s" totalImagesPullingTime="420.78086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:52.044502254 +0000 UTC m=+2609.899519405" watchObservedRunningTime="2026-10-31 03:38:52.046279651 +0000 UTC m=+2609.901296795" Oct 31 03:38:52.152389 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.152292 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:52.557667 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.557635 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:38:52.819756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:52.819668 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:53.588457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:53.582892 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:54.164951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:54.164891 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:54.881960 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:54.881859 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.567352 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:55.567311 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:56.176072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:56.176036 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:56.828606 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:56.828565 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:57.454450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:57.454381 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:58.037459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:58.037429 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:38:58.064865 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:58.064827 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:58.650395 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:58.650356 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:59.514270 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:38:59.514237 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:00.289833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:00.289791 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:01.070783 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:01.070743 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:01.666699 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:01.666664 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:02.081484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.081447 4556 generic.go:356] "Generic (PLEG): container finished" podID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerID="7478856b4df5bc1c80b867db548f246555ef18b67daa71279cd0547b1579c537" exitCode=0 Oct 31 03:39:02.081701 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.081501 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerDied","Data":"7478856b4df5bc1c80b867db548f246555ef18b67daa71279cd0547b1579c537"} Oct 31 03:39:02.081915 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.081900 4556 scope.go:122] "RemoveContainer" containerID="7478856b4df5bc1c80b867db548f246555ef18b67daa71279cd0547b1579c537" Oct 31 03:39:02.083183 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.083153 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:39:02.083250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.083226 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:39:02.083286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.083264 4556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 03:39:02.289507 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:02.289388 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:03.006008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.005976 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:39:03.055825 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.055780 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:03.095934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.095900 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerStarted","Data":"837cf6f14cd5ac5ab526d4582038f4305e8c1c6cf1375918ca7e09cc1b5fa623"} Oct 31 03:39:03.290496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.284785 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:39:03.290496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.286815 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 03:39:03.290496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.286869 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 03:39:03.793460 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:03.793400 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:04.292856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:04.292816 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 03:39:04.293265 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:04.292873 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 03:39:04.410429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:04.410382 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:04.592504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:04.590040 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:05.082438 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:05.072983 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:05.286439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:05.284834 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 03:39:05.286439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:05.284900 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 03:39:05.764589 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:05.764510 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:06.283446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:06.280777 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 03:39:06.283446 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:06.280852 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 03:39:06.465822 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:06.465782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:07.148466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:07.148404 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:07.290439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:07.289812 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.290439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:07.289891 4556 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 31 03:39:07.784280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:07.784235 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:08.300374 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:08.300331 4556 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:39:08.300956 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:08.300932 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:39:08.302782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:08.302759 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:39:08.466750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:08.466695 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:09.063250 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:09.063207 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:09.669481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:09.669439 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:10.190149 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:10.190109 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:10.391739 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:10.391686 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:10.981113 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:10.981074 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:11.574111 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:11.574054 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:12.227302 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:12.227263 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:12.506327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:12.506243 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:12.768120 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:12.767998 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:13.437245 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:13.437214 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:14.056894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:14.056837 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:14.661775 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:14.661717 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:15.371618 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:15.371587 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:15.935635 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:39:15.935591 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:39:15.990239 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:15.990177 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:16.570354 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:16.570323 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.257674 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:17.257645 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.958636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:17.958501 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:18.568828 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:18.568785 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:19.169631 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:19.169591 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:19.808126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:19.808059 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:20.362496 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:20.362445 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:21.030863 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:21.030832 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:21.653138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:21.653067 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:22.271821 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:22.271782 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:22.872574 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:22.872537 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:23.467931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:23.467894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:24.077876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:24.076456 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:24.668234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:24.668196 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:25.276323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:25.276279 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:25.950402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:25.950361 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:26.556588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:26.556541 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:27.156888 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:27.156841 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:27.752582 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:27.752515 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:28.458698 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:28.458666 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.055457 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:29.055400 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.060264 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:29.060241 4556 scope.go:122] "RemoveContainer" containerID="ecf5937e9054c52edcad93ab4f51f52ede0f113f93c29560a30df1866180fad2" Oct 31 03:39:29.657343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:29.657311 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:30.256003 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:30.255936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:30.858024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:30.857971 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:31.475205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:31.475153 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:32.223514 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:32.223483 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:32.776212 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:32.776172 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:33.360072 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:33.360027 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:33.958439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:33.958384 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:34.713326 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:34.713294 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:35.270101 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:35.270064 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:35.864649 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:35.864612 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:36.466854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:36.466802 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:37.062343 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:37.062308 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:37.674173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:37.674131 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:38.312794 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:38.312755 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.064844 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:39.064798 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.665466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:39.665384 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:40.263518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:40.263481 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:40.869355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:40.869311 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:41.463392 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:41.463351 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:42.064223 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:42.064145 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:42.663780 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:42.663746 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:43.260898 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:43.260845 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:43.873272 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:43.873204 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:44.472464 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:44.472409 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.168931 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:45.168894 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.638356 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:45.638264 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.777687 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:45.777652 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.936511 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:39:45.936477 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:39:46.367526 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:46.367342 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:46.982936 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:46.982892 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:47.567025 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:47.566987 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:48.179845 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:48.179805 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:48.893889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:48.893856 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:49.462257 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:49.462218 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:50.024047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:50.024012 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n"] Oct 31 03:39:50.026744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:50.026717 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dfc9n"] Oct 31 03:39:50.067047 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:50.066973 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:50.256377 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:50.256340 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2792d0-25a2-4088-b902-b0bd58b6f832" path="/var/lib/kubelet/pods/9e2792d0-25a2-4088-b902-b0bd58b6f832/volumes" Oct 31 03:39:50.670802 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:50.670763 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:51.371021 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:51.370978 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:51.950552 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:51.950390 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:52.569170 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:52.569138 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:53.165953 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:53.165913 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:53.761323 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:53.761286 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:54.366537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:54.366498 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:54.964767 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:54.964731 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.576391 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:55.576353 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:56.168188 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:56.168149 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:56.847015 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:56.846960 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:57.465952 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:57.465915 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:58.067437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:58.067368 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:58.668240 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:58.668189 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.339746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:59.339527 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.956368 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:39:59.956332 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:00.561068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:00.561029 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:01.175142 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:01.175108 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:01.774320 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:01.774268 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:02.528437 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:02.528389 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:03.261961 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:03.261912 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:04.074599 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:04.074545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:04.846612 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:04.846575 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:05.490234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:05.490197 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:06.300532 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:06.300498 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:06.962588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:06.962550 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:07.658971 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:07.658916 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:08.538762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:08.538724 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:09.036894 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:09.036856 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd"] Oct 31 03:40:09.038782 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:09.038750 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6pmpd"] Oct 31 03:40:09.265028 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:09.264994 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:09.879169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:09.879123 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:10.253429 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:10.253385 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:10.256700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:10.256675 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cd67c2-46ef-4a9e-920d-f41bbe7e9898" path="/var/lib/kubelet/pods/e7cd67c2-46ef-4a9e-920d-f41bbe7e9898/volumes" Oct 31 03:40:10.655173 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:10.655137 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:11.364694 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:11.364659 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.047043 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:12.047001 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.752770 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:12.752711 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:13.370737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:13.370683 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:14.083925 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:14.083888 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:14.817495 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:14.817465 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:15.472341 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:15.472290 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:15.935623 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:40:15.935586 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:40:16.113430 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:16.113393 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:16.809996 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:16.809962 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:17.655380 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:17.655338 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:18.400024 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:18.399984 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:19.025148 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:19.025113 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:19.762566 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:19.762526 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:20.472892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:20.472842 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:21.151332 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:21.151273 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:21.893913 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:21.893880 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:22.560851 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:22.560818 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.269883 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:23.269839 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.870225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:23.870190 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:24.558180 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:24.558144 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:25.567286 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:25.567246 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:26.254355 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:26.254310 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:27.068585 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:27.068552 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:27.769467 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:27.769381 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:28.477876 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:28.477824 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:29.210027 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:29.209995 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:30.004228 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:30.004188 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:30.737533 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:30.737499 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:31.407559 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:31.407523 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:31.990234 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:31.990196 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:32.668986 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:32.668938 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:33.276895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:33.276860 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:33.968484 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:33.968452 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:34.644259 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:34.644221 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:35.137867 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:35.137830 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:35.282182 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:35.282148 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:35.425982 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:35.425936 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:40.941579 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:40.941539 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:40:40.942219 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:40.941781 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" podUID="93595565-ed34-4e71-ba21-1a189f2901ef" containerName="maas-api" containerID="cri-o://ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619" gracePeriod=30 Oct 31 03:40:41.032039 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.031995 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:41.194769 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.194706 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:40:41.245957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.245926 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\") pod \"93595565-ed34-4e71-ba21-1a189f2901ef\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " Oct 31 03:40:41.245957 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.245974 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\") pod \"93595565-ed34-4e71-ba21-1a189f2901ef\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " Oct 31 03:40:41.246235 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.246111 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\" (UniqueName: \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\") pod \"93595565-ed34-4e71-ba21-1a189f2901ef\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " Oct 31 03:40:41.246306 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.246235 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\") pod \"93595565-ed34-4e71-ba21-1a189f2901ef\" (UID: \"93595565-ed34-4e71-ba21-1a189f2901ef\") " Oct 31 03:40:41.246697 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.246664 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca" pod "93595565-ed34-4e71-ba21-1a189f2901ef" (UID: "93595565-ed34-4e71-ba21-1a189f2901ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:41.247050 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.247026 4556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93595565-ed34-4e71-ba21-1a189f2901ef-openshift-service-ca\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:41.248373 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.248338 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs" pod "93595565-ed34-4e71-ba21-1a189f2901ef" (UID: "93595565-ed34-4e71-ba21-1a189f2901ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:41.248610 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.248585 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7" pod "93595565-ed34-4e71-ba21-1a189f2901ef" (UID: "93595565-ed34-4e71-ba21-1a189f2901ef"). InnerVolumeSpecName "kube-api-access-jrdt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:41.249788 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.249764 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls" pod "93595565-ed34-4e71-ba21-1a189f2901ef" (UID: "93595565-ed34-4e71-ba21-1a189f2901ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:41.352856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.349472 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdt7\" (UniqueName: \"kubernetes.io/projected/93595565-ed34-4e71-ba21-1a189f2901ef-kube-api-access-jrdt7\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:41.352856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.349510 4556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-metrics-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:41.352856 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.349536 4556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93595565-ed34-4e71-ba21-1a189f2901ef-maas-api-tls\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:41.592809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.592721 4556 generic.go:356] "Generic (PLEG): container finished" podID="93595565-ed34-4e71-ba21-1a189f2901ef" containerID="ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619" exitCode=0 Oct 31 03:40:41.592809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.592772 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" event={"ID":"93595565-ed34-4e71-ba21-1a189f2901ef","Type":"ContainerDied","Data":"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619"} Oct 31 03:40:41.592809 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.592783 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" Oct 31 03:40:41.593068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.592836 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb" event={"ID":"93595565-ed34-4e71-ba21-1a189f2901ef","Type":"ContainerDied","Data":"d7dd74b777f81e7f756419297c62ec6baf390285a718c5441d2ea2fc70bdc2bd"} Oct 31 03:40:41.593068 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.592858 4556 scope.go:122] "RemoveContainer" containerID="ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619" Oct 31 03:40:41.602824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.602803 4556 scope.go:122] "RemoveContainer" containerID="ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619" Oct 31 03:40:41.603092 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:40:41.603072 4556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619\": container with ID starting with ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619 not found: ID does not exist" containerID="ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619" Oct 31 03:40:41.603157 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.603105 4556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619"} err="failed to get container status \"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619\": rpc error: code = NotFound desc = could not find container \"ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619\": container with ID starting with ac35951d7111dc23d29cf6a7c557ff3d45e0a17711671b5b0fb0178de25db619 not found: ID does not exist" Oct 31 03:40:41.616252 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.616227 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:40:41.622554 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:41.622534 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-8d20d2-7b66fdcf64-pwhwb"] Oct 31 03:40:42.256516 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:42.256486 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93595565-ed34-4e71-ba21-1a189f2901ef" path="/var/lib/kubelet/pods/93595565-ed34-4e71-ba21-1a189f2901ef/volumes" Oct 31 03:40:42.517260 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:42.517180 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:40:42.669454 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:42.669394 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:45.935566 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:40:45.935526 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:40:48.123205 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:48.123173 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:48.370949 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:48.370910 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:48.498684 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:48.498651 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:40:48.498909 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:48.498885 4556 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" containerID="cri-o://837cf6f14cd5ac5ab526d4582038f4305e8c1c6cf1375918ca7e09cc1b5fa623" gracePeriod=30 Oct 31 03:40:48.531370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:48.531283 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:50.610966 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:50.610932 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:40:53.305284 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.305247 4556 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:40:53.306064 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.306022 4556 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:40:53.659651 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.659618 4556 generic.go:356] "Generic (PLEG): container finished" podID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerID="837cf6f14cd5ac5ab526d4582038f4305e8c1c6cf1375918ca7e09cc1b5fa623" exitCode=0 Oct 31 03:40:53.659824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.659655 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerDied","Data":"837cf6f14cd5ac5ab526d4582038f4305e8c1c6cf1375918ca7e09cc1b5fa623"} Oct 31 03:40:53.659824 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.659778 4556 scope.go:122] "RemoveContainer" containerID="7478856b4df5bc1c80b867db548f246555ef18b67daa71279cd0547b1579c537" Oct 31 03:40:53.752892 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.752867 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:40:53.793750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793716 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.793943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793786 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.793943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793855 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.793943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793881 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.793943 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793911 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.794210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.793989 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.794210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794020 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:53.794210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794030 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.794210 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794139 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.794405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794207 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:53.794405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794230 4556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\") pod \"9fc2d605-d2f7-48ed-bbba-99620489dd31\" (UID: \"9fc2d605-d2f7-48ed-bbba-99620489dd31\") " Oct 31 03:40:53.794405 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794293 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:53.794613 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794584 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:53.794676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794619 4556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-certs\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.794676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794640 4556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-credential-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.794676 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794656 4556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-data\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.794797 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.794358 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:53.796371 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.796349 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "kube-api-access-7lmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:53.796577 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.796553 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:53.796658 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.796556 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:40:53.796737 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.796719 4556 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token" pod "9fc2d605-d2f7-48ed-bbba-99620489dd31" (UID: "9fc2d605-d2f7-48ed-bbba-99620489dd31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895521 4556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-workload-socket\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895551 4556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-token\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895560 4556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2d605-d2f7-48ed-bbba-99620489dd31-istiod-ca-cert\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895569 4556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-envoy\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895579 4556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmlm\" (UniqueName: \"kubernetes.io/projected/9fc2d605-d2f7-48ed-bbba-99620489dd31-kube-api-access-7lmlm\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:53.895604 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:53.895588 4556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2d605-d2f7-48ed-bbba-99620489dd31-istio-podinfo\") on node \"ip-10-0-139-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:54.666122 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:54.666094 4556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" Oct 31 03:40:54.666563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:54.666144 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t" event={"ID":"9fc2d605-d2f7-48ed-bbba-99620489dd31","Type":"ContainerDied","Data":"634a16125659b95211938824ae572a7769cf6f306e7197e1b27ba429bdf86fef"} Oct 31 03:40:54.666563 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:54.666197 4556 scope.go:122] "RemoveContainer" containerID="837cf6f14cd5ac5ab526d4582038f4305e8c1c6cf1375918ca7e09cc1b5fa623" Oct 31 03:40:54.682479 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:54.682452 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:40:54.686881 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:54.686858 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d20d2-openshift-default-6cc78d9df7-l659t"] Oct 31 03:40:55.912551 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:55.912517 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:40:56.262466 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:40:56.262373 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" path="/var/lib/kubelet/pods/9fc2d605-d2f7-48ed-bbba-99620489dd31/volumes" Oct 31 03:41:10.311643 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:10.311557 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:11.043288 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:11.043253 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:41:12.814070 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:12.814032 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:13.015220 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:13.015169 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:15.938183 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:41:15.938146 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:41:29.097322 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:29.097282 4556 scope.go:122] "RemoveContainer" containerID="70673fad6419c437047ab41d4d3d267d9ef13b89f7b5f58a15ee3801e03d4864" Oct 31 03:41:29.108169 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:29.108142 4556 scope.go:122] "RemoveContainer" containerID="42a078052608cab3ce57f81ce50e754e89efc8d08034271c671d6eb0fcf775de" Oct 31 03:41:37.831537 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:37.831500 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-cr6cj"] Oct 31 03:41:38.164746 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:38.164715 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848575bfd6-5xvtk_8fd99a4a-9462-4af7-9469-60814b1efc22/authorino/0.log" Oct 31 03:41:45.935929 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:41:45.935867 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:41:49.676289 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:49.676255 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mmt5h_84f209d2-7d1a-4b6d-8e04-e50fc1427597/manager/0.log" Oct 31 03:41:51.600657 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:51.600626 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57d9c7bd57-knch8_2c9225c1-75ee-4b76-9e11-06df9096f2fd/maas-api/0.log" Oct 31 03:41:52.186303 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:52.186272 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-058578c2-2w4ff_a4800db2-c612-46b0-9292-e12f5e87ccf0/revoke-keys/0.log" Oct 31 03:41:52.380456 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:52.380406 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-g68kp_4afbc339-ced3-43ea-8d16-ef505603a2b9/revoke-keys/0.log" Oct 31 03:41:52.555232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:52.555153 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-s8m5p_089382a9-1b28-454f-a2bf-e5f02138f0aa/revoke-keys/0.log" Oct 31 03:41:54.704401 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:54.704368 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848575bfd6-5xvtk_8fd99a4a-9462-4af7-9469-60814b1efc22/authorino/0.log" Oct 31 03:41:54.940743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:54.940708 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8ljq2_8a5570e2-5a95-49d3-851c-7cedfdfe0bbc/manager/0.log" Oct 31 03:41:55.769987 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:55.769954 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cr6cj_4ddc1e3c-e417-4674-9c14-fa83a173aa02/limitador/0.log" Oct 31 03:41:56.767056 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:56.767026 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf_cbb6aa56-2c35-45a7-b37d-98ec927dd0ee/istio-proxy/0.log" Oct 31 03:41:57.184098 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:57.184064 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-dx65w_d839699b-3e0b-419d-bc00-8e07fe5c997a/kube-auth-proxy/0.log" Oct 31 03:41:57.552324 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:57.552243 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5bjzn_3fa8ea03-5ae9-4b78-948e-96ee4132a113/istio-proxy/0.log" Oct 31 03:41:57.978740 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:57.978715 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-gc7b2_b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb/istio-proxy/0.log" Oct 31 03:41:58.971524 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:58.971495 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-qbtsd_50da3361-2849-446a-85a7-f0d8c96ef051/istio-proxy/0.log" Oct 31 03:41:59.573300 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:59.573271 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9_23938e29-580a-4626-be9a-8218e14034c9/istio-proxy/0.log" Oct 31 03:41:59.759581 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:41:59.759554 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jkdp_b3d3cba7-e7fe-43ae-89a2-3fce419f1232/payload-pre-processing/0.log" Oct 31 03:42:01.338184 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:01.338155 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ztdt5_377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4/main/0.log" Oct 31 03:42:01.689233 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:01.689201 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-79tcd_3ef6c22d-6630-4ae2-8065-4dce4bf222c5/main/0.log" Oct 31 03:42:01.875720 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:01.875680 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4_e930e90e-f96a-4890-a4f3-0e3ebaf83a66/main/0.log" Oct 31 03:42:02.420509 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:02.420477 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd_ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205/main/0.log" Oct 31 03:42:03.405670 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:03.405625 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848575bfd6-5xvtk_8fd99a4a-9462-4af7-9469-60814b1efc22/authorino/0.log" Oct 31 03:42:10.345162 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:10.345116 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:13.567138 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:13.567112 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mmt5h_84f209d2-7d1a-4b6d-8e04-e50fc1427597/manager/0.log" Oct 31 03:42:15.248450 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:15.248406 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57d9c7bd57-knch8_2c9225c1-75ee-4b76-9e11-06df9096f2fd/maas-api/0.log" Oct 31 03:42:15.923387 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:15.923359 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-058578c2-2w4ff_a4800db2-c612-46b0-9292-e12f5e87ccf0/revoke-keys/0.log" Oct 31 03:42:15.935380 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:42:15.935350 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:42:16.120861 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:16.120832 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-g68kp_4afbc339-ced3-43ea-8d16-ef505603a2b9/revoke-keys/0.log" Oct 31 03:42:16.326762 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:16.326691 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-s8m5p_089382a9-1b28-454f-a2bf-e5f02138f0aa/revoke-keys/0.log" Oct 31 03:42:18.148596 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:18.148561 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848575bfd6-5xvtk_8fd99a4a-9462-4af7-9469-60814b1efc22/authorino/0.log" Oct 31 03:42:18.407346 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:18.407318 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8ljq2_8a5570e2-5a95-49d3-851c-7cedfdfe0bbc/manager/0.log" Oct 31 03:42:19.194728 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:19.194698 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cr6cj_4ddc1e3c-e417-4674-9c14-fa83a173aa02/limitador/0.log" Oct 31 03:42:20.004550 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.004521 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf_cbb6aa56-2c35-45a7-b37d-98ec927dd0ee/istio-proxy/0.log" Oct 31 03:42:20.025711 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.025683 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p"] Oct 31 03:42:20.028706 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.028680 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-s8m5p"] Oct 31 03:42:20.256796 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.256720 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089382a9-1b28-454f-a2bf-e5f02138f0aa" path="/var/lib/kubelet/pods/089382a9-1b28-454f-a2bf-e5f02138f0aa/volumes" Oct 31 03:42:20.425232 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.425205 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-dx65w_d839699b-3e0b-419d-bc00-8e07fe5c997a/kube-auth-proxy/0.log" Oct 31 03:42:20.830069 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:20.830038 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5bjzn_3fa8ea03-5ae9-4b78-948e-96ee4132a113/istio-proxy/0.log" Oct 31 03:42:21.308225 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:21.308183 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-gc7b2_b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb/istio-proxy/0.log" Oct 31 03:42:22.378833 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:22.378802 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-qbtsd_50da3361-2849-446a-85a7-f0d8c96ef051/istio-proxy/0.log" Oct 31 03:42:23.025327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:23.025296 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9_23938e29-580a-4626-be9a-8218e14034c9/istio-proxy/0.log" Oct 31 03:42:23.080575 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:23.080539 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:23.213481 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:23.213441 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:23.243573 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:23.243543 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jkdp_b3d3cba7-e7fe-43ae-89a2-3fce419f1232/payload-pre-processing/0.log" Oct 31 03:42:24.438280 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:24.438249 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ztdt5_377b06b5-3c1e-4455-a1ec-e3cf9b09f8d4/main/0.log" Oct 31 03:42:24.852462 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:24.852368 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-79tcd_3ef6c22d-6630-4ae2-8065-4dce4bf222c5/main/0.log" Oct 31 03:42:25.045364 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:25.045337 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8sg6w4_e930e90e-f96a-4890-a4f3-0e3ebaf83a66/main/0.log" Oct 31 03:42:25.651778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:25.651749 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9pwhd_ca0e3e54-3fc4-4bd5-9e2f-c61cdec88205/main/0.log" Oct 31 03:42:34.027013 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:34.026972 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp"] Oct 31 03:42:34.030204 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:34.030177 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g68kp"] Oct 31 03:42:34.255750 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:34.255674 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afbc339-ced3-43ea-8d16-ef505603a2b9" path="/var/lib/kubelet/pods/4afbc339-ced3-43ea-8d16-ef505603a2b9/volumes" Oct 31 03:42:45.935492 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:42:45.935451 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:42:52.026764 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:52.026724 4556 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff"] Oct 31 03:42:52.033126 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:52.031526 4556 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058578c2-2w4ff"] Oct 31 03:42:52.256008 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:42:52.255975 4556 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4800db2-c612-46b0-9292-e12f5e87ccf0" path="/var/lib/kubelet/pods/a4800db2-c612-46b0-9292-e12f5e87ccf0/volumes" Oct 31 03:43:10.451243 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:10.451210 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:11.129744 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:11.129710 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:15.935538 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:43:15.935497 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:43:29.144818 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:29.144784 4556 scope.go:122] "RemoveContainer" containerID="9d3ea914a2ad5b0d4127f70db34b5e7cd08a839e9d1217b25ec4f34fc70db7fc" Oct 31 03:43:45.934795 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:43:45.934758 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:43:55.846778 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:55.846741 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:56.016179 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:56.016148 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:59.119854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:59.119824 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j8dxn_b5bf57c6-bc1c-4f99-9e94-5f36c30f4bec/global-pull-secret-syncer/0.log" Oct 31 03:43:59.409660 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:59.409632 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" Oct 31 03:43:59.625665 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:43:59.625638 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-52.us-east-2.compute.internal_01b1495007bcae44e02a368d0e145561/haproxy/0.log" Oct 31 03:44:05.908318 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:05.908232 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848575bfd6-5xvtk_8fd99a4a-9462-4af7-9469-60814b1efc22/authorino/0.log" Oct 31 03:44:05.958753 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:05.958722 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8ljq2_8a5570e2-5a95-49d3-851c-7cedfdfe0bbc/manager/0.log" Oct 31 03:44:06.138644 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:06.138610 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-cr6cj_4ddc1e3c-e417-4674-9c14-fa83a173aa02/limitador/0.log" Oct 31 03:44:08.075141 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.075111 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-589dc7c4b9-vc6j9_4975fc72-46ce-4ecc-87eb-500108bb4304/metrics-server/0.log" Oct 31 03:44:08.273168 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.273137 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lk6f_f32d48d6-7d90-4329-86a4-6243beb7f9fd/node-exporter/0.log" Oct 31 03:44:08.304634 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.304598 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lk6f_f32d48d6-7d90-4329-86a4-6243beb7f9fd/kube-rbac-proxy/0.log" Oct 31 03:44:08.335369 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.335297 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lk6f_f32d48d6-7d90-4329-86a4-6243beb7f9fd/init-textfile/0.log" Oct 31 03:44:08.578636 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.578608 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/prometheus/0.log" Oct 31 03:44:08.611934 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.611856 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/config-reloader/0.log" Oct 31 03:44:08.642402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.642373 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/thanos-sidecar/0.log" Oct 31 03:44:08.673439 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.673400 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/kube-rbac-proxy-web/0.log" Oct 31 03:44:08.706249 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.706225 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/kube-rbac-proxy/0.log" Oct 31 03:44:08.737470 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.737443 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/kube-rbac-proxy-thanos/0.log" Oct 31 03:44:08.769100 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:08.769079 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_14b1a193-5933-4c75-a294-d15a7f16832f/init-config-reloader/0.log" Oct 31 03:44:09.165964 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.165929 4556 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp"] Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166309 4556 cpu_manager.go:414] "containerMap: removing container" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166319 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166353 4556 cpu_manager.go:414] "containerMap: removing container" podUID="93595565-ed34-4e71-ba21-1a189f2901ef" containerName="maas-api" Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166358 4556 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93595565-ed34-4e71-ba21-1a189f2901ef" containerName="maas-api" Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166407 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="93595565-ed34-4e71-ba21-1a189f2901ef" containerName="maas-api" Oct 31 03:44:09.166459 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.166434 4556 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc2d605-d2f7-48ed-bbba-99620489dd31" containerName="istio-proxy" Oct 31 03:44:09.169295 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.169276 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.179518 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.179486 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp"] Oct 31 03:44:09.256071 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.256039 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-sys\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.256254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.256158 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-lib-modules\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.256254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.256220 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-podres\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.256254 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.256252 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-proc\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.256370 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.256317 4556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwft\" (UniqueName: \"kubernetes.io/projected/cfafd62f-a0c6-46d1-954c-789c323c036e-kube-api-access-5gwft\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.265504 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.265484 4556 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68h4s\"/\"default-dockercfg-sdxsf\"" Oct 31 03:44:09.267842 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.267826 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"openshift-service-ca.crt\"" Oct 31 03:44:09.267889 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.267830 4556 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68h4s\"/\"kube-root-ca.crt\"" Oct 31 03:44:09.357914 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.357875 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-sys\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.357940 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-lib-modules\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.357956 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-podres\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.357976 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-proc\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.358002 4556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwft\" (UniqueName: \"kubernetes.io/projected/cfafd62f-a0c6-46d1-954c-789c323c036e-kube-api-access-5gwft\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.358012 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-sys\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.358053 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-podres\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.358100 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-lib-modules\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.358117 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.358117 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfafd62f-a0c6-46d1-954c-789c323c036e-proc\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.369700 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.369675 4556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwft\" (UniqueName: \"kubernetes.io/projected/cfafd62f-a0c6-46d1-954c-789c323c036e-kube-api-access-5gwft\") pod \"perf-node-gather-daemonset-fgzhp\" (UID: \"cfafd62f-a0c6-46d1-954c-789c323c036e\") " pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.483685 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.483608 4556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:09.762675 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:09.762645 4556 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp"] Oct 31 03:44:09.764466 ip-10-0-139-52 kubenswrapper[4556]: W1031 03:44:09.764438 4556 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcfafd62f_a0c6_46d1_954c_789c323c036e.slice/crio-65e58298b087e5d90761bedadf5249aa81a09527463ba483cdfd529fcecbec15 WatchSource:0}: Error finding container 65e58298b087e5d90761bedadf5249aa81a09527463ba483cdfd529fcecbec15: Status 404 returned error can't find the container with id 65e58298b087e5d90761bedadf5249aa81a09527463ba483cdfd529fcecbec15 Oct 31 03:44:10.475568 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:10.475523 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:10.609327 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:10.609291 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" event={"ID":"cfafd62f-a0c6-46d1-954c-789c323c036e","Type":"ContainerStarted","Data":"4093cbad82fdb2b45be73ba05ff8613565d48a167221b968f8b3bb3e6f41c69b"} Oct 31 03:44:10.609536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:10.609341 4556 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" event={"ID":"cfafd62f-a0c6-46d1-954c-789c323c036e","Type":"ContainerStarted","Data":"65e58298b087e5d90761bedadf5249aa81a09527463ba483cdfd529fcecbec15"} Oct 31 03:44:10.609536 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:10.609441 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:10.627196 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:10.627144 4556 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" podStartSLOduration=1.627129823 podStartE2EDuration="1.627129823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:10.625244499 +0000 UTC m=+2928.480261639" watchObservedRunningTime="2026-10-31 03:44:10.627129823 +0000 UTC m=+2928.482146961" Oct 31 03:44:11.260588 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:11.260545 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:13.777241 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:13.777214 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6x7z_ea5034e0-53b0-48d6-8d9e-71c8eb577ca4/dns/0.log" Oct 31 03:44:13.804081 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:13.804056 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6x7z_ea5034e0-53b0-48d6-8d9e-71c8eb577ca4/kube-rbac-proxy/0.log" Oct 31 03:44:13.863895 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:13.863864 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k45bw_371390b5-879d-403c-b7e5-58bbceb94ba0/dns-node-resolver/0.log" Oct 31 03:44:14.500756 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:14.500717 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d5d5b459f-d5zvx_ca616b97-e45b-49e3-8ee3-85af2a334fc3/registry/0.log" Oct 31 03:44:14.556617 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:14.556586 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wjn6d_dd57deac-c763-4eb7-9010-8d38ea97766c/node-ca/0.log" Oct 31 03:44:15.146625 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:15.146595 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.356127 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:15.356087 4556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chd4v_8246785f-965d-4107-9e6e-1bed982b3d0e/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.673152 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:15.673119 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf_cbb6aa56-2c35-45a7-b37d-98ec927dd0ee/istio-proxy/0.log" Oct 31 03:44:15.934886 ip-10-0-139-52 kubenswrapper[4556]: E1031 03:44:15.934799 4556 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkthd" containerName="extractor" Oct 31 03:44:16.624275 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:16.624251 4556 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68h4s/perf-node-gather-daemonset-fgzhp" Oct 31 03:44:19.554362 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:19.554331 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c85d847d5nqpgf_cbb6aa56-2c35-45a7-b37d-98ec927dd0ee/istio-proxy/0.log" Oct 31 03:44:19.928922 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:19.928892 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-dx65w_d839699b-3e0b-419d-bc00-8e07fe5c997a/kube-auth-proxy/0.log" Oct 31 03:44:20.003402 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:20.003374 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5bjzn_3fa8ea03-5ae9-4b78-948e-96ee4132a113/istio-proxy/0.log" Oct 31 03:44:20.069763 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:20.069732 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-gc7b2_b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb/istio-proxy/0.log" Oct 31 03:44:20.244294 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:20.244220 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-qbtsd_50da3361-2849-446a-85a7-f0d8c96ef051/istio-proxy/0.log" Oct 31 03:44:20.342132 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:20.342102 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9_23938e29-580a-4626-be9a-8218e14034c9/istio-proxy/0.log" Oct 31 03:44:20.375903 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:20.375867 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jkdp_b3d3cba7-e7fe-43ae-89a2-3fce419f1232/payload-pre-processing/0.log" Oct 31 03:44:22.273793 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:22.273766 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7ngz_191190f6-9b6d-498d-b960-67437ac0b983/serve-healthcheck-canary/0.log" Oct 31 03:44:22.786710 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:22.786678 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkthd_aef181b7-b312-4f41-b898-be29316d5638/kube-rbac-proxy/0.log" Oct 31 03:44:22.814951 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:22.814926 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkthd_aef181b7-b312-4f41-b898-be29316d5638/exporter/0.log" Oct 31 03:44:22.843293 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:22.843266 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkthd_aef181b7-b312-4f41-b898-be29316d5638/extractor/0.log" Oct 31 03:44:23.474743 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.474714 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d9fdc54f-dx65w_d839699b-3e0b-419d-bc00-8e07fe5c997a/kube-auth-proxy/0.log" Oct 31 03:44:23.516131 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.516103 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-5bjzn_3fa8ea03-5ae9-4b78-948e-96ee4132a113/istio-proxy/0.log" Oct 31 03:44:23.562854 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.562821 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-gc7b2_b29c7ebc-a98a-484f-b69f-dc57eaa6ddfb/istio-proxy/0.log" Oct 31 03:44:23.664980 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.664943 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-qbtsd_50da3361-2849-446a-85a7-f0d8c96ef051/istio-proxy/0.log" Oct 31 03:44:23.727916 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.727838 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-77ll9_23938e29-580a-4626-be9a-8218e14034c9/istio-proxy/0.log" Oct 31 03:44:23.749155 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:23.749128 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jkdp_b3d3cba7-e7fe-43ae-89a2-3fce419f1232/payload-pre-processing/0.log" Oct 31 03:44:25.682650 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:25.682618 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-57b855755-jfdnq_d82faf3e-0d6b-4118-9949-611b32936f28/manager/0.log" Oct 31 03:44:25.706543 ip-10-0-139-52 kubenswrapper[4556]: I1031 03:44:25.706518 4556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-57b855755-jfdnq_d82faf3e-0d6b-4118-9949-611b32936f28/copy-manifests/0.log"